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ABSTRACT 


This  research  focuses  on  the  application  of  existing  assessment  tools  that  may  be 
applicable  to  Marine  Air  Ground  Task  Force  (MAGTF)  Command,  Control, 
Communications  and  Computers  (C4)  System  of  Systems  (SoS)  performance  assessment 
efforts.  An  analysis  of  the  Marine  Corps  Tactical  Systems  Support  Activity’s 
(MCTSSA’s)  C4  SoS  assessment  approach  provides  a  means  for  defining  a  MAGTF  C4 
SoS  and  for  illustrating  how  that  SoS  is  represented  in  the  assessment  environment.  This 
provides  the  framework  and  context  for  follow-on  examination  of  SoS  performance 
metrics.  The  challenges  with  defining  specific  performance  metrics  and  examination  of 
past  assessment  events  using  those  metrics  provide  the  basis  for  discussion  of  alternative 
approaches  and  application  of  assessment  tools  specifically  tailored  for  SoS  assessment 
efforts.  Three  specific  tools,  i-Score,  Interoperability  Quotient  (IQ),  and  Dynamic 
Software  Architecture  Visualization  and  Evaluation  (DynSAVE),  are  examined.  The 
results  indicate  i-Score  and  DynSAVE  offer  the  greatest  potential  applicability  to  the 
MAGTF  SoS  assessment  effort.  In  a  culminating  discussion,  applying  a  multicriteria 
identification  process  to  obtain  a  mathematical  model  that  correlates  an  interoperability 
measure  with  measurable  SoS  performance  criteria  is  proposed  as  a  means  of  extending 
the  i-Score  model  for  greater  applicability  to  SoS  assessment  and  performance 
improvement  efforts. 
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EXECUTIVE  SUMMARY 


Over  a  number  of  years,  the  Marine  Corps  Tactical  Systems  Support  Activity  has 
aggressively  pursued  new  and  innovative  ways  to  improve  efforts  to  assess  the 
performance  of  tactical  C4  Systems  of  Systems  prior  to  deployment  in  an  operational 
environment.  In  that  pursuit,  determining  how  best  to  approach  performance  assessments 
of  large-scale,  complex  SoS  has  proven  a  challenge  that  extends  beyond  the  Marine 
Corps  to  a  fundamental  system  of  systems  engineering  challenge  shared  by  acquisition 
and  test  organizations  and  activities  throughout  the  Department  of  Defense.  Selecting 
key  performance  criteria,  developing  methodologies  for  conducting  large-scale  SoS 
assessments  and  defining  more  quantitative  means  for  measuring  and  assessing  SoS 
performance  and  behavior  all  serve  as  a  central  challenge  and  the  genesis  of  this  study. 

After  a  high-level  review  of  efforts  to  address  the  challenges  associated  with  SoS 
assessments  at  both  Joint  and  Marine  Corps  component  level,  three  tools  were  examined 
for  their  applicability  specifically  to  C4  SoS  assessment  efforts.  While  each  tool  provides 
a  unique  approach  that  may  help  better  quantify  the  performance  of  a  MAGTF  C4  SoS, 
they  are  also  very  similar  in  that  they  each  examine  information  exchanges  at  the 
component  level  to  derive  an  overall  SoS  performance  assessment  measure  in  terms  of 
interoperability.  The  tools  examined  included  the  i-Score  methodology  developed 
through  the  Air  Force  Institute  of  Technology,  the  Interoperability  Quotient  (IQ) 
methodology  developed  by  Northrop  Grumman  Corporation  and  the  Dynamic  Software 
Architecture  Visualization  and  Evaluation  (DynSAVE)  tool  and  process  developed  by  the 
Fraunhofer  Center  for  Experimental  Software  Engineering  Maryland. 

Of  these  tools,  i-Score  and  DynSAVE  offer  the  most  potential  for  follow-on 
analysis,  with  the  i-Score  methodology  providing  a  means  for  deriving  a  quantitative 
measure  of  SoS  interoperability  and  the  DynSAVE  software  tool  providing  a  means  of 
quantifying  the  anomalous  behavior  and  relative  efficiency  of  the  SoS.  A  discussion  of 
an  approach  to  develop  a  mathematical  model  that  correlates  an  interoperability  measure 
with  measurable  SoS  performance  criteria  using  a  multicriteria  identification  process 

serves  as  the  culminating  effort  of  this  study.  With  a  mature  mathematical  model  and 
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high  degree  of  confidence  in  the  correlation  between  the  performance  criteria  measure 
predicted  by  the  interoperability  model  and  the  performance  measure  observed  from  the 
prototype,  the  mathematical  model  may  provide  a  cost  effective  and  rapid  means  to 
examine  the  SoS  for  potential  performance  improvement. 
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INTRODUCTION 


A.  BACKGROUND 

In  warfare  past  and  present,  the  pursuit  of  improved  and  more  effective  Command 
and  Control  (C2)  on  the  battlefield  has  been,  and  is  still,  a  relentless  effort.  As  one  of  the 
Marine  Corps’  fundamental  warfighting  functions,  C2  is  unique  in  that  it  enables  all  the 
other  warfighting  functions:  intelligence,  maneuver,  logistics,  fires  and  force  protection 
(U.S.  Marine  Corps,  2001).  Through  C2,  these  functions  are  executed  with  a  coherency 
and  purpose  that  results  in  an  overall  warfighting  capability  that  is  greater  than  the  sum  of 
the  capabilities  provided  by  the  individual  warfighting  functions. 

The  proliferation  of  modem  Command,  Control,  Communications  and  Computer 
(C4)  systems  and  C4  system  of  systems  (SoS)  on  the  battlefield  now  serves  as  the  focus 
of  today’s  effort  to  improve  the  effectiveness  of  C2.  Advancements  in  technology  and 
interoperability  pursued  and  leveraged  through  C4  systems  development  efforts  have 
contributed  to  unprecedented  C2  capabilities,  allowing  once  disparate  systems  to  work 
together  in  a  collaborative  and  synergistic  manner  that  significantly  enhances  the  ability 
of  a  commander  to  manage  the  battlespace. 

However,  at  the  same  time,  the  advent  of  C4  systems  has  added  a  new  dimension 
of  complexity  to  the  battlefield.  The  C4  systems  and  SoS  are  now  an  integral  part  of  the 
battlespace,  with  the  performance  of  the  C4  SoS  very  much  interconnected  to  the 
successful  execution  of  the  warfighting  functions  in  pursuit  of  battlespace  domination. 

Furthermore,  the  dynamics  of  C4  SoS  performance  are  in  themselves  complex 
and  multidimensional.  Changes  to  an  individual  subsystem  in  the  SoS  architecture  may 
result  in  second-  and  third-order  effects  that  result  in  degradation  to  the  performance  of 
the  overall  SoS  that  may  be  unrelated  to  the  performance  of  the  subsystem  that  has 
undergone  change.  To  understand  these  complexities  and  to  ensure  that  a  modification  to 
an  existing  C4  SoS  architecture  does  not  degrade  overall  performance  of  the  SoS,  the 
Marine  Corps,  for  a  number  of  years,  has  attempted  to  establish  processes  and  procedures 
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for  conducting  C4  SoS  assessment  testing.  However,  while  methodologies  for  testing 
individual  systems  are  well  documented  and  understood,  there  are  a  number  of  unique 
challenges  associated  with  SoS  assessment  testing. 

One  of  the  most  basic  and  critical  challenges  is  determining  appropriate  testable 
performance  measures  for  the  SoS.  While  the  individual  systems  that  make  up  the  SoS 
typically  have  well  documented  performance  criteria  that  can  be  used  as  the  basis  for  an 
assessment,  in  many  cases  the  SoS  may  not.  The  challenge  with  determining  appropriate 
performance  measures  in  concert  with  many  other  complexities  associated  with  SoS 
testing  has  resulted  in  testing  approaches  that  differ  very  little  from  the  methodology  used 
to  test  the  individual  systems  that  constitute  the  SoS.  That  is  to  say,  from  a  measurement 
and  performance  perspective,  the  individual  systems  in  the  SoS  tend  to  serve  as  the  focus 
of  the  test,  which  may  not  necessarily  provide  a  complete  or  useful  assessment  of  the  SoS 
as  a  whole.  This  seems  intuitive  from  the  definition  of  a  SoS  “as  a  set  or  arrangement  of 
systems  that  results  from  independent  systems  integrated  into  a  larger  system  that  delivers 
unique  capabilities”  (DAU,  2010).  Those  unique  SoS  capabilities  imply  unique  attributes 
and  behaviors  derived  from  the  arrangement  and  integration  of  the  independent  systems. 
These  combined  capabilities,  attributes  and  behaviors  uniquely  characterize  the  SoS  and 
imply  that  performance  measures  beyond  the  performance  measures  associated  with  the 
independent  systems  are  needed  to  assess  the  SoS. 

B.  PURPOSE 

In  the  Office  of  the  Under  Secretary  of  Defense  (Acquisition,  Technology  and 
Logistics)  publication,  Systems  Engineering  Guide  for  Systems  of  Systems  (2008),  a 
number  of  conditions  are  described  that  contribute  to  the  challenges  associated  with 
Systems  of  Systems  engineering.  With  Systems  of  Systems  often  comprised  of 
individual  systems  in  different  stages  of  a  very  dynamic  and  often  asynchronous 
acquisition  life  cycle  management  process  and  crossing  many  organizational 
management  boundaries  within  the  DoD  and  commercial  industry,  there  is  a  need  to 
expand  and  redefine  systems  engineering  processes  to  accommodate  these  needs  (Office 
of  the  Deputy  Under  Secretary  of  Defense  for  Acquisition  and  Technology  [OSD],  2008): 
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Beyond  these  development  challenges,  depending  on  the  complexity  and 
distribution  of  the  constituent  systems,  it  may  be  infeasible  or  very 
difficult  to  completely  test  and  evaluate  SoS  capabilities.  (OSD,  2008, 
p.  13) 

While  the  Department  of  Defense  (DoD)  and  systems  engineering  community 
continue  to  refine  and  mature  the  processes  for  SoS  engineering  and  development,  the 
systems  engineering  testing  community  has  struggled  with  defining  how  best  to  measure 
and  assess  the  performance  of  the  SoS.  From  that  struggle,  various  test  tools  and  test 
methodologies  have  been  developed  and  demonstrated  to  address  the  test  and  evaluation 
challenge.  This  thesis  investigates  a  small  subset  of  existing  assessment  tools  and 
analysis  concepts  that  may  be  extended  to  the  Marine  Corps’  C4  System  of  Systems 
assessment  methodology  as  a  means  to  obtain  a  more  holistic  assessment  of  the  SoS 
under  evaluation. 

C.  RESEARCH  QUESTIONS 

Test  and  Evaluation  (T&E)  activities  span  the  continuum  of  a  disciplined  systems 
engineering  approach.  As  the  engineering  community  attempts  to  understand  and  deal 
with  the  complexities  of  large-scale  system  of  systems  development  efforts,  the  need  to 
apply,  adapt  and  modify  T&E  activities  to  accommodate  a  SoS  engineering  approach  are 
evident.  The  following  questions  were  designed  to  understand  the  challenges  with  SoS 
testing  and  opportunities  for  addressing  those  challenges  through  specific  tools  and 
analysis  processes.  While  the  questions  are  generic  in  nature,  the  intent  is  to  address 
them  in  context  with  the  specific  challenges  associated  with  the  Marine  Corps’  approach 
to  SoS  assessment  testing. 

1 .  How  can  large-scale  C4  System  of  Systems  be  tested  and  evaluated  in  a 
manner  that  reflects  performance  attributes  associated  with  the  System  of 
Systems  as  a  whole? 

2.  What  are  the  key  attributes  of  a  C4  SoS  that  may  serve  as  the  basis  for  SoS 
level  performance  criteria? 

3.  What  are  some  existing  assessment  tools  and  how  may  they  be  extended  to  aid 
in  C4  SoS  assessment  process? 
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4.  How  might  multicriteria  identification  improve  or  contribute  to  the  Marine 
Corps’  C4  SoS  assessment  methodology? 

D.  BENEFITS  OF  STUDY 

As  the  Marine  Corps  continues  to  refine  the  processes  and  techniques  for  C4 
system  of  systems  assessment,  other  DoD  organizations  at  both  Joint  and  Component 
levels  are  doing  the  same.  This  thesis  is  intended  to  serve  as  a  basis  of  knowledge  that 
can  be  leveraged  by  the  Marine  Corps  and  other  DoD  test  and  evaluation  activities, 
improving  the  use  of  assessment  tools  and  analysis  concepts  to  address  the  challenges 
associated  with  test  and  evaluation  of  large  scale  and  complex  C4  system  of  systems. 

E.  SCOPE  AND  METHODOLOGY 

This  thesis  focuses  on  the  application  of  existing  assessment  tools  and  analysis 
concepts  that  may  be  applicable  to  MAGTF  C4  SoS  performance  assessment  efforts. 
Data  and  documentation  from  past  C4  SoS  test  events  are  used  as  well  as  input  from  the 
Marine  Corps  test  and  evaluation  team  planning  future  test  events. 

An  analysis  of  the  Marine  Corps  Tactical  Systems  Support  Activity’s 
(MCTSSA’s)  C4  SoS  assessment  approach,  while  serving  in  a  lead  role  for  the  Marine 
Corps’  SoS  assessment  effort,  provides  a  means  for  defining  a  MAGTF  C4  SoS  and  for 
illustrating  how  that  SoS  is  represented  in  the  assessment  environment.  This  provides  the 
framework  and  context  for  follow-on  examination  and  discussion  of  performance  metrics 
that  are  necessary  for  characterizing  and  ultimately  assessing  the  performance  of  the  SoS. 
The  challenges  with  defining  specific  performance  metrics  and  examination  of  past 
assessment  events  using  those  metrics  provide  the  basis  for  discussion  of  alternative 
approaches  and  application  of  assessment  tools  specifically  tailored  for  SoS  assessment 
efforts. 

Three  specific  tools  are  examined  and  viewed  relative  to  their  applicability  to  the 
MAGTF  SoS  assessment  effort.  The  tools  include  i-Score,  an  Air  Force  Institute  of 
Technology  (AFIT)  initiative  to  measure  SoS  interoperability;  Interoperability  Quotient 
(IQ),  a  commercial  application  demonstrated  by  the  Navy  to  measure  SoS  performance; 
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and  Software  Architecture  Visualization  and  Evaluation  (SAVE),  a  toolset  developed 
through  the  Fraunhofer  Institute  of  Technology  for  optimizing  and  analyzing  the 
architecture  of  implemented  software  systems. 

Building  on  the  examination  of  available  analysis  tools,  an  examination  of  how  a 
Multicriteria  Identification  analysis  concept  may  be  applied  to  a  MAGTF  C4  SoS  and 
extended  as  a  means  to  improve  SoS  assessment  testing  is  investigated.  This  culminates 
in  an  overall  recommendation  presented  in  the  final  chapter  for  improving  C4  SoS 
assessments  through  appropriate  use  of  assessment  tools  and  analysis  concepts.  Overall 
methodology  is  depicted  in  Figure  1. 


Figure  1.  Study  Methodology. 
Performance  Tool  Assessment  Methodology 
Process  that  guided  this  research. 
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II.  MAGTF  C4  SOS  TEST  AND  EVALUATION  PROGRAM 

REVIEW 


A.  INTRODUCTION 

The  Marine  Corps  Systems  Command  (MCSC)  serves  as  the  Marine  Corp’s  agent 
for  acquisition  and  sustainment  of  systems  and  equipment.  MCTSSA  is  a  subordinate 
command  within  MCSC,  providing  engineering  and  technical  services  in  support  of 
MCSC’s  acquisition  and  sustainment  role.  Within  the  MCSC  organization,  MCTSSA 
falls  under  the  cognizance  of  the  Deputy  Commander  for  Systems  Engineering, 
Interoperability,  Architectures  &  Technology  (DC  SI  AT),  who  also  serves  as  the  Marine 
Corps’  Technical  Authority  Deputy  Warranting  Officer.  SIAT  is  chartered  with 
providing  “enterprise  level  system  engineering  across  product  lines  and  product  life 
cycles  to  ensure  MCSC  provides  end-to-end  integrated,  interoperable,  and  certified 
warfighting  capabilities”  (SIAT,  2010).  In  effect,  SIAT  serves  as  an  agent  for  enabling 
horizontal  integration  across  the  MCSC  product  groups  and  systems  development  efforts. 
The  resulting  goal  is  to  bring  the  individual  systems  together  into  a  coherent,  integrated 
system  of  systems  architecture  to  provide  warfighting  capabilities. 

In  December  2006,  the  DC  SIAT  initiated  an  effort  to  develop  a  capability 
certification  process  to  assess  MCSC  developed  systems  within  a  MAGTF  SoS 
environment.  MCTSSA,  with  a  long  history  of  supporting  C4  systems  throughout  the 
acquisition  life  cycle,  was  tasked  to  develop  the  processes  for  assessing  a  MAGTF  C4 
SoS  in  support  of  the  overall  certification  process  for  the  MAGTF  C2  capability. 

Assessing  a  MAGTF  C4  SoS  was  not  new  to  MCTSSA.  Previous  attempts  in  the 
form  of  a  Federation  of  Systems  (FEDOS)  test  and  evaluation  program  had  met  with 
mixed  results.  While  FEDOS  brought  various  C4  systems  into  a  SoS  environment  for 
testing,  no  objective  SoS-level  performance  requirements  were  established.  With  no 
established  SoS-level  performance  requirements,  the  individual  component  systems  in  the 
SoS  became  the  default  focus  of  the  test  effort  with  performance  metrics  of  the  individual 
component  systems  in  the  SoS  determined  through  negotiation  with  product  sponsors. 

The  product  sponsors  were  suspect  of  the  overall  test  results  and  questioned  how  the  test 
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agency  employed  the  individual  system  in  the  C4  architecture  (logical  and  physical 
layout  of  the  SoS  test  environment).  Further,  they  failed  to  see  value  of  a  test  that  went 
beyond  the  scope  of  Developmental  and  Operational  Test  requirements  as  offered  by 
Acosta  et  al.  (2007): 

Further,  FEDOS  was  perceived  as  a  no-win  situation  for  Product  Groups. 

After  a  system  had  successfully  passed  Development  and  Operational 
Tests,  and  demonstrated  compliance  with  system-level  performance 
requirements  as  described  in  the  Capability  Development  Document 
(CDD),  Operational  Requirements  Document  (ORD),  or  equivalent, 

FEDOS  tested  component  systems  in  ways  they  had  not  been  designed  to 
be  used.  (Acosta  et  al.,  2007,  p.  29) 

The  Marine  Corps’  current  SoS  assessment  effort  that  resulted  from  the  2006 
directive  from  DC  SI  AT,  termed  MAGTF  C4I  Capabilities  Certification  (MC3).  The 
MC3  effort  is  built  upon  the  lessons  learned  from  the  FEDOS  program  and  leverages 
efforts  in  the  Joint  test  community  looking  to  establish  methods  for  executing  tests  of  SoS 
in  the  Joint  mission  environment. 

B.  MAGTF  C4  SOS  DEFINED 

Developing  a  common  definition  for  a  System  of  Systems  (SoS)  has  been  an 
evolving  process  and  the  subject  of  numerous  discussion  papers  in  industry  and  the  DoD. 
In  an  IEEE  International  Conference  on  Systems  of  Systems  Engineering  2009  article, 
John  Clark  provides  a  simple  definition  of  a  SoS  as  “The  sum  of  the  whole  is  greater  than 
the  sum  of  the  individual  parts”  in  which  the  parts  are  integrated  and  may  or  may  not  be 
members  of  a  common  domain  (2009,  p.  2). 

In  the  DoD,  this  definition  is  expanded  to  accommodate  considerations  unique  to 
the  warfighting  domain.  The  basic  idea  of  the  sum  of  the  whole  being  greater  that  sum  of 
the  individual  parts  is  still  present,  but  the  definition  now  also  infers  association  with  a 
warfighting  capability.  In  the  Chairman  of  the  Joint  Chiefs  of  Staff  Instruction,  CJCSI 
3 170.0 IF  glossary,  the  SoS  is  defined  as: 
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A  set  or  arrangement  of  interdependent  systems  that  are  related  or 
connected  to  provide  a  given  capability.  The  loss  of  any  part  of  the  system 
could  significantly  degrade  the  performance  or  capabilities  of  the  whole. 

The  development  of  an  SoS  solution  will  involve  trade  space  between  the 
systems  as  well  as  within  an  individual  system  performance.  (CJCS,  2007, 
p.  GL-19) 

This  SoS  definition  is  aligned  with  the  Joint  Capabilities  Integration  and 
Development  System  (JCIDS)  that  serves  as  an  overarching  process  that  guides  the 
development  of  new  capabilities  for  DoD  Military  Forces.  JCIDS  provides  a 
methodology  that  is  intended  to  improve  the  DoD  acquisition  process  through  a  level  of 
governance  over  the  individual  Service’s  acquisition  efforts  with  a  focus  on  identifying 
shortcomings  and  redundancy  from  a  Joint  warfighting  capabilities  perspective. 

However,  while  JCIDS  provides  the  acquisition  community  with  a  more  holistic 
and  SoS  approach  to  the  Joint  acquisition  process,  the  DoD  Planning,  Programming, 
Budgeting,  and  Execution  (PPBE)  process  that  guides  acquisition  funding  and  program 
management  is  not  specifically  structured  to  address  the  complexities  of  SoS  acquisition 
efforts.  That,  along  with  internal  acquisition  community  politics,  legacy  acquisition 
policies  and  many  other  factors  has  contributed  to  an  acquisition  approach  that  is  focused 
on  the  individual  system  in  terms  of  equipping  the  warfighter  with  a  material  solution 
supporting  a  needed  capability.  A  system  is  engineered  and  developed  to  provide  a 
function  or  capability  within  the  context  of  being  employed  within  a  larger  SoS  (physical 
interfaces  and  data  exchanges  with  other  systems  indentified  and  defined),  but  the  SoS 
itself  is  more  of  a  cooperative  instantiation  than  an  engineered  entity. 

However,  as  the  definition  and  understanding  of  the  SoS  concept  matures,  the 
engineering  focus  in  the  acquisition  process  is  beginning  to  transform.  A  new  focus  on 
SoS  engineering  takes  systems  engineering  processes  to  the  next  level.  In  August  2008, 
The  Director,  Systems  and  Software  Engineering  Deputy  Under  Secretary  of  Defense 
(Acquisition  and  Technology)  published  Version  1  of  the  Systems  Engineering  Guide  for 
Systems  of  Systems.  Recognizing  the  growing  interdependencies  of  systems  to  provide 
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warfighting  capabilities,  the  guide  serves  as  a  first  step  to  assist  the  systems  engineering 
community  with  adapting  systems  engineering  processes  to  systems  of  systems 
engineering  needs  (OSD,  2008). 

The  guide  defines  a  SoS  through  reference  to  the  2008  DoD  Defense  Acquisition 
Guidebook  (DAG),  as  “a  set  or  arrangement  of  systems  that  results  when  independent 
and  useful  systems  are  integrated  into  a  larger  system  that  delivers  unique  capabilities” 
that  remains  unchanged  in  the  2010  update  to  the  DAG.  While  this  definition  is  not 
necessarily  novel,  the  guide  takes  a  step  further  by  defining  four  types  of  SoS:  Virtual, 
Collaborative,  Acknowledged  and  Directed,  as  depicted  in  Table  1. 


SoS  Type 

Description 

Virtual 

No  central  management  authority  or  centrally  agreed  upon  purpose  for  the  SoS  but 

displays  large  scale  emergent  behavior 

Collaborative 

Systems  interact  voluntarily  to  fill  agreed  upon  central  purpose 

Acknowledged 

Recognized  objectives,  designated  manager  and  resources  with  individual  systems 

retaining  independent  ownership,  objectives,  funding  ,  development  and  sustainment 

Directed 

Integrated  SoS  built  and  centrally  managed  for  a  specific  purpose  with  component 

systems  retaining  an  ability  to  operate  independently  but  with  independent  operational 

mode  subordinated  to  SoS  purpose 

Table  1.  Types  of  SoS. 

Definitions  of  different  types  of  System  of  Systems  as  discussed  in  the  2008  Systems 
Engineering  Guide  for  Systems  of  Systems  (After:  OSD,  2008,  pp.  4-5). 


The  Systems  Engineering  Guide  for  Systems  of  Systems  recognizes  that  most 
systems  today  are  linked  through  net-centric  information  sharing  to  form  a  Virtual  SoS. 
Collaborative  SoS  result  as  greater  interdependencies  are  recognized  and  communities  of 
interest  are  formed  to  work  together  for  mutual  benefit.  Acknowledged  SoS  are 
becoming  more  predominant  in  DoD  acquisition,  demonstrating  collaborative  behavior, 
but  with  greater  management  authority  and  resources  at  the  SoS  level.  Directed  SoS  are 
less  predominant.  The  Army’s  Future  Combat  System  (FCS)  is  offered  as  an  example  of 
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a  Directed  SoS,  with  individual  systems  development  driven  by  the  common  objectives 
of  the  SoS  (OSD,  2008).  With  budgetary  and  technology  challenges  that  resulted  in  the 
cancellation  of  the  FCS  program  in  2009,  the  Directed  SoS  may  be  the  most  challenging 
from  an  acquisition  and  management  approach  due  to  the  inherent  size  and  scope  of  a 
development  effort  at  the  Directed  SoS  level. 

In  the  Marine  Corps,  the  Combat  Operations  Center  (COC)  and  the  Common 
Aviation  Command  and  Control  System  (CAC2S)  are  two  programs  of  interest  from  a 
SoS  perspective.  The  COC  is  a  fielded  program  of  record  that  provides  the  C2  facilities 
for  the  commander  and  staff  of  the  four  components  of  the  MAGTF;  Command  Element 
(CE),  Ground  Combat  Element  (GCE),  Air  Combat  Element  (ACE),  and  Logistics 
Combat  Element  (LCE).  The  COC  consists  of  physical  components  (tentage,  power 
generation  and  environmental  controls),  Information  technology  infrastructure  (servers 
and  data  storage),  network  and  communications  management  capabilities  (network 
management  tools  and  interfaces  for  organic  communications  assets),  and  tactical 
software  applications  and  associated  computer  platforms  (PM  MAGTF  C2,  2009).  While 
some  of  the  component  systems  are  unique  to  the  COC,  the  C2-enabling  capabilities  are 
largely  derived  from  components  that  are  incorporated  within  the  COC  SoS  but  retain 
independent  ownership,  objectives,  funding  and  sustainment.  The  Advanced  Field 
Artillery  Tactical  Data  System  (AFATDS),  Command  Post  of  the  Future  (CPOF),  Joint 
Automated  Deep  Operations  Coordination  System  (JADOCS)  and  Global  Combat 
Support  System  (GCSS)  are  a  few  examples  of  systems  that  are  part  of  the  COC  baseline 
configuration  but  are  managed  as  independent  programs.  Further,  the  wide -band 
communications  infrastructure  necessary  to  interface  and  tie  individual  COC  together  is 
also  comprised  of  separately  managed  independent  programs.  Based  on  that,  the  COC,  as 
seen  in  Figure  2,  would  most  closely  represent  an  Acknowledged  SoS. 
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Figure  2.  Combat  Operations  Center  (COC)  OV-1. 

The  COC  provides  C2  facilities  for  the  components  of  the  MAGTF 
(From:  Zapata,  2010a). 


The  Marine  Corps  CAC2S  program  depicted  in  Figure  3  represents  a  more 
ambitious  approach.  The  initial  effort  focused  on  a  complete  replacement  for  C2 
equipment  of  the  Marine  Air  Command  and  Control  System  (MACCS).  With  an  intent 
to  replace  legacy  single -mission  systems  with  an  integrated  hardware  and  software 
solution  for  more  effective  command,  control  and  coordination  of  air  operations,  this 
effort  was  most  closely  aligned  with  a  Directed  SoS  development  approach.  However, 
much  like  FCS,  CAC2S  faced  program  management  and  technical  challenges  that 
required  the  program  to  restructure  in  2009.  With  that  restructure,  CAC2S  is  now  linked 
to  the  COC  program,  leveraging  the  C2  capabilities  provided  by  the  COC  to  meet  some 
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of  the  C2  needs  of  the  MACCS  mission.  Subsequently,  with  that  dependency  on  another 
SoS,  CAC2S  would  now  also  be  considered  an  Acknowledged  SoS. 


Figure  3.  Common  Aviation  Command  and  Control  System  (CAC2S)  OV-1. 
The  CAC2S  represents  an  incremental  SoS  development  effort  that  will  replace  the 
current  Marine  Air  Command  and  Control  System.  In  Increment  I,  Phase  I,  the 
functionality  of  the  Direct  Air  Support  Center  (DASC)  is  replaced  by  CAC2S  (From: 

Zapata,  2010b). 


In  the  Draft  7  January  2010  Mission  Area  “Systems  of  Systems”  Systems 
Engineering  Guidebook,  the  concept  of  mission  capability  is  reinforced: 

In  the  future,  global  operations  will  be  conducted  by  distributed,  integrated 
and  interoperable  forces.  This  future  warfare  is  about  capability  delivered 
by  ‘System  of  Systems’  operating  as  a  single  system.  System  of  Systems 
(SoS)  is  defined  in  this  document  as  a  force  package  of  interoperable 
platforms  and  nodes  acting  as  a  single  system  to  achieve  a  mission 
capability,  i.e.  a  mission  level  SoS.  Typical  characteristics  include  a  high 
degree  of  collaboration  and  coordination,  flexible  addition  or  removal  of 
component  systems,  and  a  net-centric  architecture.  The  capabilities 
provided  by  each  constituent  system  operating  within  the  SoS  are  framed 
by  the  integrated  force  package  architecture.  (Chief  Systems  Engineer, 

2010,  p.  1) 
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For  the  Marine  Corps,  the  MAGTF  serves  as  the  foundational  structure  that 
provides  a  unique  combined  arms  air-ground  warfighting  capability.  As  depicted  in 
Figure  4,  each  MAGTF  is  comprised  of  four  core  elements:  the  Command  Element  (CE) 
that  provides  for  the  overarching  command  of  the  MAGTF;  the  Ground  Combat  Element 
(GCE)  with  infantry,  artillery  and  armor;  the  Aviation  Combat  Element  (ACE)  with  fixed 
wing,  rotary  wing  and  aviation  command  and  control;  and  the  Logistics  Combat  Element 
(LCE)  with  supply,  combat  engineers,  medical  and  other  combat  services  capabilities. 
The  basic  premise  is  that  this  structure  provides  a  mission  capability  that  is  greater  than 
the  sum  of  the  individual  components:  the  very  essence  of  a  SoS. 


Figure  4.  MAGTF  Components. 

Graphical  representation  of  MAGTF  hierarchical  structure. 


While  every  MAGTF  retains  this  basic  structure,  the  size  and  individual 
component  structure  of  the  MAGTF  is  mission  dependent  and  can  range  from  largest — 
Marine  Expeditionary  Force  (MEF)  level  (approximately  45,000  personnel) — to  the 
smallest,  Marine  Expeditionary  Unit  (MEU)  level  (approximately  2,500  personnel). 
Both  COC  and  CAC2S  support  this  dynamic  structure  through  modular  and  scalable 
design.  The  COC  in  particular  is  fielded  in  five  different  capability  sets  that  are  equated 
to  an  echelon  of  command;  MEF  COC  (Capability  Set  I),  Division  COC  (Capability  Set 
II),  Regimental  COC  (Capability  Set  III),  Battalion  COC  (Capability  Set  IV)  and 
Company  COC  (Capability  Set  V).  A  MEF  level  MAGTF  with  a  MEF  headquarters, 

Marine  Division,  Marine  Aircraft  Wing  and  Logistics  Support  Group  would  then  consist 
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of  multiple  COCs  in  all  five  Capability  Set  variants.  The  CAC2S  is  fielded  in  a  similar 
scalable  manner  with  four  different  variations  to  accommodate  C2  requirements  for 
specific  MACCS  functions  at  Wing,  Squadron,  Echelon  and  Sub-Agency  levels. 

From  a  warfighting  capability  perspective,  we  can  view  the  MAGTF  and  all  its 
people,  processes  and  technology  as  an  independent  SoS,  fully  capable  of  executing 
warfighting  mission  capabilities  in  support  of  defined  operational  objectives.  The 
MAGTF  SoS  can  also  be  considered  an  element  of  larger  Joint  SoS  when  employed  as 
part  of  a  Joint  Task  Force. 

MAGTF  C4  can  be  considered  a  SoS  (within  context  of  the  greater  MAGTF  SoS) 
that  is  specifically  architected  to  deliver  a  C2  capability  in  support  of  the  C2  warfighting 
function.  MAGTF  C4  as  a  SoS  is  comprised  of  people  (C4  operators  and  maintainers), 
processes  (guided  by  doctrine  and  written  procedures)  and  technology  (communications 
systems,  networking  systems  and  C2  applications)  that  are  brought  together  through  an 
integrated  MAGTF  architecture  that  enable  unique  C2  functions  and  capabilities  that 
cannot  be  achieved  individually.  Both  COC  and  CAC2S  (or  currently  MACCS)  SoS,  are 
components  of  the  larger  MAGTF  C4  SoS,  and  together  comprise  a  significant  part  of  the 
MAGTF  C4  SoS,  with  the  intent  of  providing  C2  capabilities  at  all  echelons  of  command 
down  through  the  Company  level. 

This  is  precisely  the  concept  that  is  presented  in  the  Marine  Corps’  MAGTF  C2 
vision.  In  2008,  a  MAGTF  C2  Transition  Task  Force  was  established  to  provide 
governance  and  oversight  to  the  process  of  developing  and  fielding  systems  that  support 
C2  capability.  With  the  intent  of  harmonizing  the  development  and  fielding  of  C2 
programs  of  record  like  CAC2S,  COC  and  many  others  to  deliver  an  end-to-end 
integrated  C2  solution,  this  effort  provides  the  Marine  Corps  with  a  means  for 
transitioning  from  an  Acknowledged  to  a  more  Directed  SoS.  Cloninger  (2009)  describes 
the  mid-term  goal  of  MAGTF  C2  as  an  integrated  SoS: 

MAGTF  C2  will  become  an  integrated  C2  solution  that  will  migrate  the 
current  multiplicity  of  stove-piped,  disparate  systems  into  an  integrated 
system-of-systems  that  will  support  deployed  aspects  of  Marine  Corps  C2 
requirements  from  pre-deployment  planning  to  execution  and 
redeployment  via  multi-functional  C2  nodes,  (p.  103) 
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From  this  perspective,  MAGTF  C2  is  seen  as  the  end  state  of  the  transition  of 
MAGTF  C4  SoS  from  an  Acknowledged  to  a  MAGTF  C2  Directed  SoS.  However,  until 
that  end-state  is  achieved,  the  MAGTF  C4  SoS  (as  depicted  in  Figure  5)  then  is 
considered  an  Acknowledged  SoS,  comprised  of  both  SoS  and  individual  systems 
centrally  employed  and  managed  through  the  MAGTF  (as  a  hierarchical  organization) 
with  a  recognized  objective  to  provide  a  C2  capability  that  maximizes  combat  capability 
through  unity  of  effort. 
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Figure  5.  MAGTF  C4  SoS. 

The  MAGTF  C4  SoS  provides  a  C2  capability  to  the  MAGTF  SoS  that  in  concert  with 
the  Army  and  Air  Force  Component  SoS  make  up  the  Joint  Task  Force  SoS. 


C.  JOINT  LEVEL  SOS  TEST  AND  EVALUATION  APPROACH 

While  individual  systems  development  with  associated  developmental  test  and 
operational  test  and  evaluation  (OT&E)  is  still  the  predominant  focus  in  the  DoD 
acquisition  community,  the  reality  of  net-centric  SoS  warfare  and  the  relevance  of  SoS 
performance  to  the  ability  to  achieve  mission  capabilities  have  not  gone  unaddressed. 
Significant  efforts,  both  individual  and  collaborative,  to  test,  measure  and  quantify  the 
performance  of  complex  SoS  have  been,  and  continue  to  be,  a  major  focus  of  both 
Component  and  Joint  DoD  testing  organizations.  At  the  Joint  level,  this  effort  has 
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predominantly  fallen  under  the  purview  of  the  Under  Secretary  of  Defense  /  Acquisition 
Technology  &  Logistics  (AT&L)  and  Director  Operational  Test  &  Evaluation  (DOT&E). 

AT&L  established  the  Joint  Mission  Environment  Test  Capability  (JMETC)  in 
2006  to  provide  the  infrastructure  and  common  environment  for  distributed  Live,  Virtual 
and  Constructive  (LVC)  testing  in  the  Joint  DoD  community.  JMETC  enables 
distributed  facilities  to  link  together  through  the  existing  DoD  Secure  Defense  Research 
and  Engineering  Network  (SDREN)  to  provide  a  Joint  environment  in  support  of 
“Developmental  testing,  Operational  Testing,  Interoperability  Certification,  Net-Ready 
Key  Performance  Parameters  (KPP)  compliance  testing,  and  Joint  Mission  Capability 
Portfolio  Testing”  (Lockhart  &  Ferguson,  2008,  p.  161).  JMETC  offers  a  corporate 
approach  to  Joint  testing,  providing  readily  available  connectivity  over  existing  DoD 
networks,  a  common  middleware  for  connecting  distributed  facilities,  data  exchange 
standards,  data  management,  reuse  repository  and  a  suite  of  test  tools  for  test  planning, 
execution  and  analysis  as  described  by  (Lockhart  &  Ferguson,  2008): 

•  Connectivity.  The  Secure  Defense  Research  and  Engineering  Network 
(SDREN)  provides  connectivity  between  test  facilities.  Training  facilities  can 
also  be  linked  through  the  Joint  Training  and  Experimentation  Network 
(JTEN). 

•  Middleware.  The  Test  and  Training  Enabling  Architecture  (TENA)  serves  as 
common  data  exchange  environment  used  by  distributed  facilities  to  send  and 
receive  data.  TENA  provides  gateways  to  connect  to  other  data  exchange 
protocols  to  include  Distributed  Interactive  Simulation  (DIS)  and  High-Level 
Architecture  (HLA). 

•  Data  Exchange  Standards.  Through  TENA  Object  Models,  a  common 
language  is  provided  for  data  exchange  between  systems  in  the  distributed 
architecture. 

•  Data  Management.  JMETC  provides  a  capability  to  archive  test  data  from 
multiple  facilities  for  analyses  and  evaluation  of  test  results. 
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•  Reuse  Repository.  A  Web  portal  provides  a  means  to  share  object  models, 
middleware,  software  tools  and  documentation  across  the  user  community. 

•  Test  Tools.  A  common  suite  of  test  tools  is  provided  for  planning, 
configuration,  monitoring,  control  and  analysis.  The  Interoperability  Test  and 
Evaluation  Capability  (InterTEC)  is  a  secondary  initiative  under  AT&L  that 
started  as  a  joint  effort  between  the  Joint  Interoperability  Test  Command 
(JITC)  and  Naval  Air  Warfare  Center  (NAVAIR)  and  is  now  an  integral  part 
of  JMETC  and  synonymous  with  the  common  test  tool  suite.  From  the  JITC 
Web  site,  the  InterTEC  tool  suite  also  provides  the  synthetic  battlespace 
environment  that  shapes  the  test  scenario  through  many-on-few  scenario 
simulations  and  a  simulation/emulation  gateway  to  stimulate  system  sensors 
in  the  systems  under  test  (“JITC,”  2009). 

In  an  effort  complimentary  to  JMETC’ s  Joint  test  infrastructure  and  common 
LVC  test  environment,  the  DOT&E  Joint  Test  (JTEM)  effort  was  chartered  in  2006  for 
three  years,  to  develop,  test  and  evaluate  methods  and  processes  for  conducting  LVC 
Joint  test  events.  The  principal  product  that  resulted  from  this  effort  was  the  JTEM 
Capability  Test  Methodology  (CTM)  that  provides  a  framework  for  a  capabilities-based 
approach  for  testing  in  a  Joint  mission  environment.  CTM  version  2.0  describes  a  six- 
step  methodology,  consisting  of  14  JTEM  processes,  as  depicted  in  Figure  6. 


18 


14E 

TIE 

Ihrifri 

[TES* 

PliTi 

|TESF| 

Devckp  C  jpabilrtjrSoS 
Desciiplimn 

Develop  Joint  Operational 
Context  lor  Test  IJOC  O 
Develop  Evaluation  Strategy 
DevdcpReline:  iapabi  sly 
Cros-swa  ft 


Capability 
Set 

.Focus 


h-fl  r.y-Jn—  p*  Ij  Jui  . 


ftndyit  Data 

Evaluate  SoS  Performance  & 
Joint  MhHtm£tt«&vcri«3. 


Figure  6.  JTEM  CTMv2.0. 

The  six-step  JTEM  Capability  Test  Methodology  offers  a  structured  approach  for 
conducting  an  SoS  assessment  (From:  Fiebrandt  &  Dryer,  2009,  p.  3). 


The  six  steps  of  the  CTM  provide  a  basic  framework  for  any  SoS  evaluation. 
Outlined  in  JTEM’s  Action  Officer’s  Handbook  for  Testing  in  a  Joint  Environment 
(Lorenzo,  2009),  the  CTM  steps  include: 

•  Develop  T&E  Strategy.  This  step  was  added  in  CTMv2.0  to  address  a  critical 
first  step  to  define  the  overall  SoS  evaluation  strategy.  The  key  aspects  of  this 
step  are  to  define  the  SoS  in  terms  of  capability  and  define  the  operational 
context  for  SoS  employment. 

•  Characterize  Test.  Once  the  SoS  is  defined,  determining  what  aspects  or 
attributes  of  the  SoS  can  and  should  be  measured  to  assess  the  SoS 
performance  is  a  second  key  step.  Without  defined  SoS  measures  of 
performance  (typical  of  a  non-Directed  SoS),  this  can  present  the  most 
challenging  aspect  of  the  methodology. 
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•  Plan  Test.  The  test  plan  that  results  from  this  step  documents  the  overall  test 
strategy,  test  design,  and  data  collection  methodology  that  guides  test 
execution. 

•  Implement  LVC  Distributed  Environment.  JMETC  facilitates  implementation 
of  the  LVC  distributed  environment. 

•  Manage  Test  Execution.  The  InterTEC  tool  suite  facilitates  test  management 
and  execution. 

•  Evaluate  Capability.  The  InterTEC  tool  suite  facilitates  data  collection  across 
the  LVC  distributed  environment,  however  the  significance  of  the  evaluation 
is  largely  dependent  on  how  well  the  performance  measures  were  defined  in 
the  second  step. 

In  2008,  ITEM  demonstrated  CTM  version  2.0  in  the  FCS  Joint  Battlespace 
Dynamic  Deconfliction  (JBD2)  test  event.  The  event  provided  an  opportunity  for  both 
JTEM  and  JMETC  to  mature  their  products  and  offered  FCS  an  opportunity  to  conduct  a 
complex  SoS  test  event  in  support  of  an  acquisition  Milestone  C  decision.  Hutchison, 
Lorenzo  and  Bryan  (2009)  describe  the  complexity  of  the  event: 

To  achieve  these  goals,  JBD2  established  a  complex  joint  mission 
environment  composed  of  16  test  sites  and  more  than  40  unique  live, 
virtual,  and  constructive  systems  connected  across  four  time  zones.  These 
test  sites  represented  all  four  Services  and  the  U.S.  Joint  Forces 
Command.  Of  these  16  JBD2  sites,  10  were  reused  from  two  previous  test 
venues  that  were  a  part  of  a  series  of  events  culminating  in  JBD2.  Seven 
Service  and  joint  initiatives  were  included  as  part  of  the  test  architecture. 

JBD2  truly  provided  joint  context  and  stressed  the  boundaries  of  a  live, 
virtual,  and  constructive  joint  mission  environment,  (p.  32) 

For  the  FCS  JBC2  event,  the  SoS  under  test  provided  a  C2  capability  that  was 
focused  on  two  factors: 

•  Battlespace  Management  Capability.  Test  to  determine  whether  there  was  a 
difference  between  current  and  future  SoS  implementation  during  execution 
of  mission  tasks  during  the  mission-based  test  scenario. 
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•  Timeliness  of  C2  processes.  Test  to  determine  difference  between  current 
procedural-based  processes  versus  future  expedited-based  processes  (LeSueru, 
Millich  &  Stokes,  2009). 


Of  significance  is  that  this  event  demonstrated  test  and  evaluation  of  both  material 
(SoS  physical  configuration  with  respect  to  battlespace  management  capability)  and  non¬ 
material  (C2  processes)  aspects  of  the  SoS.  Also  relevant  is  the  use  of  a  complex,  but 
operationally  realistic  mission  scenario  as  depicted  in  Figure  7,  which  provided  the  basis 
for  stressing  the  SoS  under  test. 
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Figure  7.  JBD2  Operational  View. 

Operationally  realistic  mission  thread  from  Joint  Battlespace  Dynamic  Deconfliction  Test 
Event  (From:  LeSueru,  Millich  &  Stokes,  2009,  p.  3). 


D.  COMPONENT  (MARINE  CORPS)  SOS  TEST  &  EVALUATION 
APPROACH 

At  the  Component  level,  the  Marine  Corps  has  taken  a  similar  approach  in  both 
past  (FEDOS)  and  current  (MC3)  SoS  assessment  efforts.  Both  efforts  employed  a 
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structured,  process  driven  methodology  and  both  were  conducted  in  a  test  environment 
that  attempted  to  replicate  a  MAGTF  C4  SoS  architecture.  Although  developed  prior  to 
and  independent  of  ITEM,  initial  meeting  with  the  JTEM  program  manager  in  January 
2007  revealed  significant  parallels  in  approach  and  methodology: 

•  SoS  capability-based  assessment  approach 

•  SoS  architecture  defined  and  rooted  in  individual  systems’  JCIDS  artifacts 
(Operational  Views  and  System  Views) 

•  Operational  scenario  and  mission  tasks  provide  basis  for  evaluating  the 
performance  of  the  SoS 

•  Employed  distributed  test  architecture  with  centralized  test  control 

The  significant  similarities  in  methodology  for  SoS  testing  at  the  Joint  level  to  the 
methodology  developed  by  MCTSSA  for  SoS  testing  at  the  Component  level,  indicated 
potential  for  mutual  benefit  through  continued  collaboration.  In  October  2007,  MCTSSA 
joined  the  JMETC  community  and  began  participation  in  JMETC  test  events  as  the 
Marine  Corp  component  (MAGTF  node)  in  the  Joint  test  architecture.  This  served  as  an 
opportunity  for  the  Marine  Corps  to  further  mature  its  SoS  assessment  methodology  and 
leverage  the  capabilities  provided  by  the  JMETC  InterTEC  tool  suite.  The  collaboration 
also  highlighted  a  number  of  shared  challenges:  specifically  the  development  of 
meaningful  test  threads,  selection  of  SoS  performance  metrics  and  development  of  a 
comprehensive  and  consistent  manner  for  characterizing  the  results  of  the  SoS 
performance  assessment. 

E.  GAP  ANALYSIS  (ASSESSMENT  ISSUES  AND  DEFICIENCIES) 

The  three  shared  challenges  associated  with  SoS  assessment  (test  threads, 
performance  metrics  and  assessment  reporting)  are  closely  interrelated  and  improvements 
in  one  area  could  potentially  yield  improvement  in  the  other  areas.  From  a  Component 
(MAGTF  C4  SoS)  perspective,  a  more  detailed  discussion  of  these  challenges  follows. 
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1.  Operational  Test  Thread  Concept 

A  central  feature  of  the  Marine  Corps’  C4  SoS  testing  methodology  is  the  use  of 
MAGTF-based  capability  mission  tasks  executed  in  an  operational  scenario.  Translating 
the  mission  tasks  to  an  operationally  relevant  test  thread  has  been  a  challenge  for  a 
number  of  reasons: 

•  Each  MAGTF  is  unique.  While  each  MAGTF  consists  of  the  same  core 
elements  (CE,  GCE,  ACE  and  LCE),  the  size  and  specific  component  make 
up  varies  as  dictated  by  Mission  requirements. 

•  Each  MAGTF  C4  architecture  is  unique.  Implementation  of  the  C4 
components  in  the  MAGTF  architecture  to  provide  the  C2  capability,  though 
guided  by  doctrine  and  best  practices,  is  left  to  the  discretion  of  the  MAGTF 
commander. 

•  Execution  of  mission  tasks  can  vary.  Mission  tasks  are  well  defined  but  how 
individual  C4  components  are  used  in  concert  with  the  execution  of  the  tasks 
is  often  not  defined. 

All  these  factors  come  into  play  during  the  development  of  a  test  thread  that 
replicates  execution  of  a  mission  task  in  the  test  environment.  Assumptions  to 
accommodate  all  these  factors  are  required  as  the  test  agency  translates  the  mission  tasks 
into  a  form  that  can  be  executed  in  the  test  scenario:  mission  task  translated  to  test  thread. 
Once  developed,  the  abstracted  test  thread  must  then  be  validated  to  ensure  it  is  still 
operationally  relevant.  During  the  C4  SoS  assessment,  significant  resources  (personnel 
and  equipment)  may  be  required  to  execute  the  test  thread  depending  on  degree  of  human 
to  machine  interaction  necessary  to  complete  the  mission  task. 

To  provide  a  true  representation  of  the  MAGTF  C4  SoS  in  an  operational  context, 
a  variety  of  mission  tasks  running  in  parallel  and  at  various  stages  of  completion  are 
required.  Multiple  mission  tasks,  executed  in  concert  with  a  defined  MAGTF  battle 
rhythm,  would  provide  a  meaningful  context  for  assessing  the  performance  of  the  C4 
SoS.  However,  the  effort  and  resources  required  to  develop,  validate  and  execute 

sufficient  test  threads  to  represent  this  is  considerable  and  possibly  infeasible  without  the 
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aid  of  modeling  and  simulation.  Further,  even  with  the  MAGTF  C4  SoS  well  defined 
and  stressed  through  execution  of  realistic  mission  tasks,  determining  what  to  measure 
from  a  C4  SoS  performance  perspective,  is  still  an  issue  without  documented  mission- 
task  performance  metrics  to  test  against. 

2.  SoS  Performance  Metrics 

In  response  to  the  lack  of  defined  mission  task  performance  metrics,  the  Marine 
Corps’  2007  MC3  MAGTF  C4  SoS  effort  developed  a  unique  metrics  model  in 
collaboration  with  stakeholders  within  the  Marine  Corps  Combat  Development 
Command  (MCCDC),  SIAT,  MCTSSA,  Marine  Corps  Operational  Test  and  Evaluation 
Agency  (MCOTEA)  and  Space  and  Naval  Warfare  (SPAWAR)  Systems  Center,  Atlantic. 
The  metrics  model  was  intended  to  focus  on  characterizing  the  performance  of  the  SoS  in 
terms  of  mission-based  test  thread  execution,  vice  performance  of  individual  C4  systems 
and  was  comprised  of  five  areas  of  measurement: 

•  Operator  Complexity.  Measure  of  level  of  difficulty  to  execute  a  test  thread 
from  a  user’s  perspective.  The  measure  was  based  on  the  total  number  of 
operator  steps  required  at  each  system  or  node  in  the  SoS  during  execution  of 
the  test  thread. 

•  System  Timeliness.  Measure  of  total  system  time  to  execute  the  test  thread. 
This  measure  was  based  on  total  digital  transmission  time  and  system 
processing  time  at  each  system  or  node  in  the  SoS  during  execution  of  the  test 
thread. 

•  System  Accuracy.  Measure  of  application  layer  accuracy  as  indicated  by 
percent  completion  of  digital  messages  between  each  system  and  node  in  the 
SoS  during  execution  of  the  test  thread. 

•  System  Reliability.  Measure  of  number  of  failures  at  each  system  or  node  in 
the  SoS  during  execution  of  the  test  thread. 

•  Anomalous  Behavior.  Not  a  measure  but  a  means  to  capture  emergent  or 
unexpected  behavior  within  the  SoS  during  execution  of  the  test  thread. 
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Although  successfully  demonstrated  in  a  final  test  event,  the  value  of  the  model 
was  limited.  While  the  intent  was  to  establish  a  metrics  model  that  focused  on  SoS  level 
mission  thread  execution,  the  resulting  SoS  performance  assessment  was  extrapolated 
from  the  performance  of  the  individual  systems  in  terms  of  the  metrics  as  they  completed 
tasks  during  execution  of  the  test  thread.  Based  on  the  author’s  personal  observations 
and  involvement  with  the  2007  test  event,  this  shifted  focus  away  from  the  SoS 
performance  to  a  study  of  performance  of  individual  systems  operating  within  a  SoS. 

The  MAGTF  C2  working  group  provided  another  metrics  model.  In  the  MAGTF 
C2  Test  and  Evaluation  Master  Plan  for  2010,  five  Critical  Operational  Issues  (COIs)  are 
defined  that  guide  the  selection  of  Key  Performance  Parameters  (KPPs)  with  associated 
Measures  of  Performance  (MOPs): 

1.  To  what  level  does  MAGTF  C2  enable  Blue  Force  Tracking  of 
friendly  assets? 

2.  To  what  level  does  MAGTF  C2  provide  force  and  unit 
commanders  a  Common  Tactical  Picture  /  Common  Operational 
Picture? 

3.  To  what  level  does  MAGTF  C2  provide  adequate  Situational 
Awareness  to  unit  commanders  and  their  forces? 

4.  To  what  level  does  the  MAGTF  C2  provide  planning  and 
collaborative  functionality  in  the  net-centric,  service-oriented 
environment  to  distributed  COCs? 

5.  To  what  level  does  MAGTF  C2  provide  the  capability  to 
communicate  while  on  the  move?  (MAGTF  C2,  2008,  pp.  21-23) 

Because  the  MAGTF  C2  effort  is  primarily  an  SoS  engineering  oversight  effort 
for  a  number  of  existing  programs  of  record  (CAC2S  and  COC  serving  as  the  primary 
programs  of  record),  the  KPPS  and  MOPs  associated  with  COIs’  1,2,3  and  5  were 
extracted  from  the  existing  programs  of  record  Capability  Development  Documents 
(MAGTF  C2,  2008).  COI  4  required  creation  of  new  MOPs  aligned  with  the  Net-Ready 
KPP  defined  in  CJCSI  6212. 01E.  The  Joint  Interoperability  Test  Command  (JITC)  is 
responsible  for  conducting  interoperability  evaluations  of  programs  of  record  to  certify 
compliance  with  the  five  elements  of  the  NR-KPP: 
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1.  Compliant  solution  architecture:  technical  exchange  of  information 
and  end-to-end  use  of  that  exchange. 

2.  Compliance  with  net-centric  data  and  services  strategies; 
evaluation  of  net-centric  data  and  services  to  determine  if  net- 
ready. 

3.  Compliance  with  applicable  GIG  Technical  Guidance  (GTG). 

4.  Compliance  with  DOD  Information  Assurance  (IA)  requirements 

5.  Compliance  with  supportability  requirements  to  include  spectrum 
utilization  and  information  bandwidth  requirements,  Selective 
Availability  Anti-Spoofing  Module  (SAASM)  and  the  Joint 
Tactical  Radio  System  (JTRS),  as  applicable.  (CJCSI  6212. 01E, 

2008,  p.  A-2) 

For  the  MAGTF  C2  effort,  the  NR-KPP  certification  requirements  for  the 
individual  programs  of  record  within  the  SoS  (including  COC  and  CAC2S)  were  adapted 
to  address  NR-KPP  compliance  from  a  SoS  perspective.  This  methodology  differs 
significantly  from  the  approach  taken  in  MC3.  While  MC3  attempts  to  characterize  the 
performance  of  the  SoS  in  terms  of  mission  thread  execution  (warfighting  capability),  the 
MAGTF  C2  approach  is  more  oriented  towards  assessing  the  C2  capability  provided  by 
the  SoS. 


3.  Reporting  SoS  Performance  Results 

To  characterize  and  quantify  the  performance  of  the  C4  SoS,  MCTSSA’s  MC3 
effort  applied  a  risk-based  reporting  methodology.  During  the  assessment  event, 
mission-based  test  threads  were  executed  through  the  C4  SoS  and  data  was  collected 
based  on  the  established  areas  of  measurement  defined  for  the  event  (operator 
complexity,  system  timeliness,  system  accuracy,  system  reliability  and  anomalous 
behavior).  The  data  was  summarized  and  evaluated  by  a  panel  of  stakeholders  and 
experts  to  derive  an  overall  risk  score  associated  with  executing  the  test  thread  in  the  C4 
SoS.  The  overall  results  of  the  assessment  effort  were  reported  in  a  standard  5x5  risk 
chart  in  a  manner  similar  to  that  depicted  in  Figure  8. 
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MAGTF  C4  SoS  Mission  Thread  Execution  Performance 
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Figure  8.  MC3  Risk-based  SoS  Performance  Assessment  Reporting. 
MCTSSA’s  MC3  SoS  Assessment  risk  chart  depicts  risk  associated  with  executing  a 
given  test  thread  within  a  defined  SoS  architecture. 


Without  defined  performance  criteria  associated  with  execution  of  the  test  thread, 
the  interpretation  of  the  data  to  determine  a  risk  measure  was  subjective  and  served  only 
as  a  characterization  of  risk  associated  with  execution  of  the  test  thread.  Performance 
data  obtained  from  the  individual  systems  during  execution  of  the  test  thread  was  used  to 
develop  a  measure  of  risk  associated  with  the  success  of  executing  a  test  thread  in  the 
SoS.  The  overall  risk  associated  with  executing  Test  Thread  D  (Figure  8)  in  the  C4  SoS, 
for  example,  is  moderate  with  a  low  likelihood  of  failure,  but  catastrophic  consequence  to 
the  mission  when  it  fails.  Through  this  methodology  an  overall  assessment  of  the 
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performance  of  the  SoS  could  be  obtained  by  taking  an  average  or  weighted  average  of 
the  total  test  thread  risk  scores  to  provide  an  overall  C4  SoS  risk  score  for  use  as  the  SoS 
performance  metric. 

The  intent  was  that  this  method  would  provide  a  level  of  abstraction  that  would  be 
more  acceptable  to  the  key  stakeholders  (Program  Offices  with  ownership  of  individual 
programs  within  the  SoS),  than  a  typical  pass/fail  grading  criteria  normally  associated 
with  formal  test  activities.  While  this  reporting  methodology  does  serve  that  purpose  and 
does  provide  a  means  to  characterize  the  overall  SoS  performance  in  terms  of  mission 
task  execution,  other  key  aspects  of  the  C4  SoS;  net-centric  C4  SoS  effectiveness  and 
suitability  attributes  like  reliability  and  timeliness  measures  used  to  obtain  the  risk 
measure  for  thread  execution  in  the  SoS  are  obscured. 

4.  DoD  and  Joint  Approaches  to  the  Challenges 

In  the  2008  Systems  Engineering  Guide  for  Systems  of  Systems,  a  more 
operational  user-centric  approach  to  addressing  the  challenge  of  SoS  is  recommended: 

Because  acknowledged  SoS  typically  comprise  existing  (often  fielded) 
systems  (e.g.,  AOC,  SIAP,  MILSATCOM),  data  from  operations  is  an 
important  source  of  understanding  the  state  of  the  SoS.  Because  the  SoS 
will  likely  evolve  based  on  incremental  changes  in  individual  systems,  it  is 
important  to  have  a  set  of  user-oriented  metrics  that  can  be  applied  in 
different  settings  over  time.  The  SoS  systems  engineer  uses  data  from 
these  settings  to  analyze  SoS  performance  and  behavior;  hence,  the 
metrics  should  include  measures  that  use  data  from  operations.  These  SoS 
metrics  should  also  be  traceable  to  the  capability  objectives  established  for 
the  SoS,  and  there  may  even  be  a  need  to  rank  the  metrics  by  importance. 

These  metrics  should  not  change  as  the  capability  of  the  SoS  matures 
unless  the  capability  objectives  themselves  change.  They  must  remain 
applicable  as  the  SoS  matures  to  assess  whether  the  changes  made  are 
actually  translating  into  better  user  support.  When  captured  in  an 
operational  environment,  metrics  allow  an  independent  view  to  assess  SoS 
performance  from  the  user’s  perspectives,  and  allow  assessment  of  the 
impacts  of  external  factors  on  capability  objectives.  These  operational 
user-based  performance  assessments  do  not  substitute  for  the  technical 
reviews  and  assessments  performed  during  the  process  of  upgrading  the 
systems  in  the  SoS.  (OSD,  2008,  p.  44) 
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This  guidance  implies  greater  reliance  on  users  to  define  performance  metrics  of 
the  SoS  during  employment  in  an  operational  context.  As  an  approach  recommended  for 
an  Acknowledged  SoS,  this  would  be  very  applicable  to  a  MAGTF  C4  SoS,  however  the 
measured  attributes  of  the  SoS  (which  are  not  defined  by  this  guidance)  would  need  to  be 
carefully  selected  to  accommodate  the  inherent  variations  in  MAGTF  C4  SoS 
implementation.  User  provided  data  on  network  loading  and  battle-rhythm  induced 
variation  in  network-data  traffic  patterns  would  be  directly  applicable  to  MAGTF  C4  SoS 
testing:  providing  operationally  realistic  base-line  data  that  can  be  used  to  stress  the  SoS 
during  assessment. 

When  JTEM  reached  the  end  of  its  three-year  charter  in  2009,  the  effort  to 
continue  work  towards  SoS  test  and  evaluation  continued  at  the  Joint  level  through  a 
DOT&E  Special  Project:  Joint  Test  and  Evaluation  Methodology  -  Transition  (JTEM-T). 
Chartered  through  2011,  JTEM-T  continues  to  refine  the  JTEM  CTM  concept  with 
specific  focus  on  decomposing  “...Joint  Mission  Threads  (JMT)  into  mission-  and 
tasked-based  testable  measures”  (Walters,  2010,  p.  3).  The  JTEM-T  effort  established  a 
metrics  working  group  with  the  goal  of  developing  by  the  end  of  Fiscal  Year  2010: 

•  A  repeatable  process  for  decomposing  JMTs  into  testable  measures 

•  Actual  testable  measures  for  selected  JMTs 

•  A  report  to  DOT&E  on  recommended  changes  to  the  Joint 
Capabilities  Integration  and  Development  Systems  to  facilitate 
testing  in  a  joint  environment 

•  A  report  through  DOT&E  to  the  OUSD/AT&L  TRMC 
recommending  the  tools  and  instrumentation  needed  to  support  the 
use  of  JMT  testable  measures.  (Walters,  2010,  p.  4) 

The  significance  of  this  is  the  promise  of  a  database  of  mission-based  test  threads  and 
associated  metrics  for  warfighting  capability-based  SoS  assessment  at  the  Joint  and 
Component  level.  These  test  threads  may  also  be  adapted  for  MAGTF  C4  SoS 
assessment,  providing  stimulus  and  another  measure  of  performance  during  C4  SoS 
assessment. 
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At  the  Joint  level,  progress  has  been  demonstrated  towards  maturation  of  the 
infrastructure  (through  JMETC),  methodology  (through  JTEM)  and  initial  mission-  and 
tasked-based  testable  measures  (through  JTEM-T)  necessary  for  Joint  SoS  test  and 
evaluation.  While  these  products  may  be  appropriate  for  adaptation  by  the  Marine  Corps 
for  a  MAGTF  SoS  assessment,  they  do  not  provide  a  complete  solution  for  assessing  the 
MAGTF  C4  SoS.  At  the  JTF  or  MAGTF  level,  the  mission  task  itself  serves  as  focus  of 
assessment  as  a  measure  of  the  overall  warfighting  capability  (how  well,  how  quickly, 
how  efficiently  the  SoS  prosecutes  the  mission).  The  MAGTF  C4  SoS  contributes 
towards  the  overall  warfighting  capability  by  providing  a  C2  capability  and,  from  that 
perspective,  metrics  associated  with  assessing  the  SoS  in  terms  of  providing  the  C2 
capability  (in  line  with  the  MAGTF  C2  SoS  assessment  approach)  would  seem  the  most 
appropriate.  However  there  is  also  value  in  assessing  the  performance  in  terms  of 
mission  thread  execution  and  net-centric  C4  SoS  effectiveness  and  suitability  (in  line 
with  the  MC3  effort)  (Figure  9). 


A  complete  MAGTF  C4  SoS  assessment  should  include  metrics  related  to  net-centric 
effectiveness  and  suitability,  mission  thread  execution  and  C2  services. 
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Documenting  and  reporting  the  results  of  the  SoS  performance  assessment  in  a 
comprehensive,  consistent  and  useful  manner  continues  to  be  a  challenge.  This  challenge 
is  due  in  part  the  fluidity  of  the  SoS  assessment  approach  (defining  test  threads  and 
metrics)  but  is  also  due  to  the  complex  nature  and  broad  scope  associated  with 
characterizing  the  behavior  of  a  large  scale  SoS.  Characterizing  a  SoS  in  terms  of  risk 
that  may  be  decreased  or  increased  through  a  change  in  SoS  baseline  configuration  may 
have  some  value.  Other  approaches,  such  as  characterizing  the  SoS  in  terms  of 
interoperability,  may  also  have  value  and  are  the  subject  of  a  number  of  independent 
efforts  in  DoD  and  the  commercial  sector. 

F.  SUMMARY 

As  the  definition  of  a  SoS  within  a  DoD  operational  context  has  evolved  and 
matured,  so  have  the  engineering  approaches  to  developing  and  assessing  a  SoS.  At  the 
Joint  SoS  level,  the  warfighting  capability  serves  as  the  focus  of  assessment,  enabled  by 
the  JMETC  distributed  SoS  testing  infrastructure,  InterTEC  test  tool  suite,  JTEM  process 
and  procedures  and  JTEM-T  Joint  Mission  Threads  (JMTs).  This  Joint  SoS  assessment 
methodology  can  be  leveraged  for  Component  SoS  assessment  (MAGTF  as  the  Marine 
Corps  Component)  by  adapting  appropriate  mission  threads  to  the  component  warfighting 
capabilities. 

For  the  MAGTF  C4  SoS,  the  C2  capability  that  enables  the  C2  warfighting 
function  provides  the  focus  for  SoS  assessment.  Executing  mission-based  test  threads 
within  a  MAGTF  C4  SoS  that  is  stressed  through  an  operationally  realistic  variable 
network  load  and  data  traffic  patterns,  provides  an  appropriate  foundation  for  an 
assessment.  Determining  the  specific  attributes  of  the  SoS  to  measure  continues  to  be  an 
evolving  process.  For  the  Marine  Corps’  MAGTF  C4  SoS,  a  blend  of  a  number  of 
approaches  may  provide  a  more  holistic  assessment  of  the  SoS.  With  an  assessment  of 
mission  thread  execution  (timeliness,  completion  percentage),  C2  services  (situational 
awareness,  position  reporting,  collaboration,  and  other  Defense  Information  Systems 


31 


Network  (DISN)  services)  and  net-centric  C4  SoS  effectiveness  and  suitability 
(bandwidth  efficiency,  network  reliability),  a  more  comprehensive  measure  of  SoS 
performance  can  be  obtained. 

As  the  techniques,  processes  and  procedures  for  executing  SoS  assessments 
mature,  there  is  an  increasing  need  to  convey  the  assessment  results  in  a  context  that 
provides  a  more  useful  and  meaningful  characterization  of  the  SoS  performance.  In  the 
next  sections  a  select  sampling  of  commercial  tools  and  techniques  will  be  examined  that 
may  aid  in  this  effort. 
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III.  CURRENT  ASSESSMENT  TOOLS  WITH  POTENTIAL 
APPLICABILITY  TO  MARINE  CORPS  C4  SOS  PERFORMANCE 

ASSESSMENT 


A.  INTRODUCTION 

The  Marine  Corps  C4  SoS  performance  assessment  effort  contributes  to  ensuring 
that  the  fielded  C4  SoS  provides  an  end-to-end  integrated  C2  solution  for  the  Marine 
Corps  Operating  Forces.  Optimally  this  assessment  would  address  all  aspects  of  the  SoS: 
people  (C4  operators  and  maintainers),  processes  (guided  by  doctrine  and  written 
procedures)  and  technology  (communications  systems,  networking  systems  and  C2 
applications).  However,  in  that  respect,  even  the  assessment  of  the  smallest  MAGTF 
(MEU),  would  require  considerable  resources  to  include  as  many  as  2500  operators.  In 
practice  then,  the  SoS  assessment  out  of  practical  necessity  must  be  conducted  through  a 
modeled  abstraction  of  a  MAGTF  C4  SoS.  Typically,  the  representative  MAGTF  C4 
SoS  designed  for  the  assessment  is  a  sampling  of  equipment  and  software  applications 
sufficient  for  executing  select  mission-based  test  threads  and  providing  sufficient 
interactions  to  invoke  the  SoS  behavior  necessary  to  provide  a  basis  for  performance 
measurement.  Furthermore,  the  assessment  focuses  primarily  on  the  technology  aspects 
of  the  SoS  (the  material  solution  for  C2)  to  minimize  the  need  for  end-user  operators  and 
maintainers  and  lessen  impact  to  the  Marine  Corps  Operating  Forces.  Other  key  aspects 
of  the  SoS  (both  people  and  processes)  serve  as  acknowledged  constraints  to  the  conduct 
of  the  SoS  assessment  and  require  assumptions  regarding  operator  training  level  and 
fidelity  of  the  processes  associated  with  the  SoS.  While  these  aspects  are  integral  to  the 
overall  SoS  performance,  a  more  narrow  focus  on  the  technical  aspects  of  the  MAGTF 
C4  SoS  provides  a  more  granular  evaluation  of  this  aspect  than  typically  afforded  by  a 
formal  MCOTEA  Operational  Test  and  Evaluation  capability-based  test  of  effectiveness 
and  suitability. 

Even  with  this  focused  approach  and  corresponding  reduction  in  scope  and 
complexity,  characterizing  the  performance  of  the  SoS  in  terms  of  the  criteria  associated 
with  the  technology  is  a  challenge  in  and  of  itself.  MCTSSA’s  MC3  assessment  effort 
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attempts  to  do  this  through  a  composite  of  various  performance  metrics  to  produce  an 
overall  risk-based  assessment  characterization  of  the  SoS.  This  approach,  while  relating 
SoS  performance  to  execution  of  a  mission-based  test  thread  and  providing  some  level  of 
traceability  of  the  thread  execution  to  the  contributions  of  individual  systems  within  the 
SoS,  has  only  been  demonstrated  in  the  Marine  Corps  MC3  effort  with  relatively  small 
numbers  of  systems  (5-10)  in  the  SoS  test  framework.  To  assess  a  SoS  during  execution 
of  more  complex  mission-based  test  threads  or  simultaneous  execution  of  multiple  test 
threads  that  involve  significantly  more  systems  as  well  as  invoke  more  complex  network 
behaviors,  the  need  for  assessment  tools  becomes  evident.  In  addition  to  helping  with 
management  of  greater  quantities  of  data,  assessment  tools  can  help  with  standardizing 
the  methodology  for  reporting,  weighting  and  summing  the  SoS  metric  data  for  an  overall 
SoS  performance  assessment.  A  sampling  of  some  assessment  tools  that  attempt  to  do 
just  that  is  examined  below. 

B.  ASSESSMENT  TOOLS 

A  theme  common  to  a  number  of  the  tools  proposed  for  SoS  assessment  is  the 
idea  of  using  interoperability  as  an  overall  SoS  performance  measure.  However,  much 
like  the  definition  of  SoS,  interoperability  must  also  be  defined  in  a  relevant  context.  The 
DoD  Dictionary  of  Military  and  Associated  Terms  defines  interoperability  as: 

1 .  The  ability  to  operate  in  synergy  in  the  execution  of  assigned  tasks. 

2.  The  condition  achieved  among  communications-electronics 
systems  or  items  of  communications-electronics  equipment  when 
information  or  services  can  be  exchanged  directly  and 
satisfactorily  between  them  and/or  their  users.  (Joint  Chiefs  of 
Staff,  2009) 

The  first  definition  aligns  nicely  with  the  basic  definition  and  concept  of  a  SoS, 
the  second  with  the  focus  on  the  technology  aspects  of  a  SoS  during  an  assessment  event. 
Both  definitions  would  then  seem  to  imply  that  interoperability  could  serve  as  a  focus  for 
assessing  overall  SoS  performance. 

In  2004,  the  DoD  contracted  Carnegie  Mellon  Software  Engineering  Institute  to 
examine  the  concept  of  interoperability  and  define  a  SoS  interoperability  (SOSI)  model. 
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The  model  described  in  their  final  report  incorporates  aspects  of  technical 
interoperability,  operational  interoperability  and  programmatic  interoperability.  The 
SOSI  model  draws  on  technical  and  operational  interoperability  concepts  introduced  by 
the  Levels  of  Information  System  Interoperability  (LISI)  model,  NATO  C3  Technical 
Architecture  (NC3TA)  Reference  Model  for  Interoperability,  the  Organizational 
Interoperability  Maturity  Model  (OIM)  and  Levels  of  Conceptual  Interoperability  Model 
(LCIM).  The  idea  of  programmatic  interoperability  is  introduced  as  another  dimension 
of  interoperability  by  the  SOSI  model  to  recognize  the  influence  that  acquisition  activities 
have  on  achieving  the  aspects  of  interoperability  (Morris,  Levine,  Meyers,  Place  & 
Palakosh,  2004).  Different  levels  of  sophistication  of  SoS  interoperability  are  central  to 
all  the  interoperability  models  and  closely  parallel  the  concept  of  different  types  of  SoS 
(Virtual,  Collaborative,  Acknowledged  and  Directed)  defined  in  Systems  Engineering 
Guide  for  Systems  of  Systems  (OSD,  2008).  The  technical  and  operational  aspects  of 
interoperability  also  closely  parallel  aspects  of  how  we  view  a  SoS  from  an  assessment 
perspective.  This  further  strengthen  the  argument  for  using  a  measure  of  interoperability 
as  a  means  of  assessing  overall  SoS  performance. 

1.  i-Score 

Developed  by  Major  Thomas  Ford,  Dr.  John  Colombi,  Dr  Scott  Graham  and  Dr 
David  Jacques  from  the  Air  Force  Institute  of  Technology  (AFIT),  the  Interoperability 
Score  (i-Score)  model  is  intended  to  offer  a  more  quantitative  measure  of  assessing  the 
interoperability  of  a  SoS  than  the  qualitative  measure  (levels  of  interoperability 
sophistication)  provided  by  models  like  LISI,  OIM  and  SOSI  (Ford,  Colombi,  Graham  & 
Jacques,  2007).  The  model  asserts  a  number  of  strengths  to  include: 

1 .  It  is  easily  computed, 

2.  It  is  based  upon  an  operational  thread, 

3.  It  makes  use  of  existing  architecture  data, 

4.  It  can  be  used  for  scenarios  where  one  or  more  type  of 
interoperability  is  represented  (i.e.,  information  and  organizational 
interoperability) 
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5.  It  defines  the  optimum  interoperability  for  a  given  operational 
thread  allowing  a  decision  maker  to  understand  what  the  limits  of 
his/her  interoperability  improvements  are  and  what  can 
realistically  be  done  to  improve  interoperability  for  an  operational 
interest  and 

6.  It  provides  a  means  of  drilling  down  into  a  process  to  discover 
where  interoperability  problems  lie.  (p.  2) 

The  model  presents  a  mathematical  means  for  measuring  the  interoperability  of  a 
SoS  obtained  by  analyzing  the  SoS  components  that  contribute  to  the  execution  of  a 
mission  test  thread.  A  mathematical  score  (i-Score  measure)  is  calculated  based  on  the 
values  attributed  to  the  interactions  between  components  (value  of  -1,0  or  1  assigned 
depending  on  degree  of  data  translation  needed  between  components)  and  the  number  of 
times  the  component  is  used  during  execution  of  the  mission  thread.  The  i-Score 
measure  is  obtained  through  a  six  step  methodology  summarized  by  Ford,  Colombi, 
Graham  and  Jacques  (2006)  as  follows: 

Step  1  -  Diagram  the  operational  thread  (e.g.,  time-critical-targeting) 
using  an  IDEFO,  BPML,  or  UML  activity  diagram  and  define  the  ordered 
set  T  of  systems  supporting  each  activity  in  the  thread. 

Step  2  -  Create  an  interoperability  matrix  M  =  [c,5;,  Jra„  where  n  is  the 

number  of  systems  supporting  the  thread,  C  =  [c(/  ]  is  a  multiplicity 

matrix  which  describes  the  number  of  times  a  system  is  used  in  the  thread, 
and  S  =  [sv  j  ^  is  a  spin  matrix  where  sip  s{- 1,0,1},  is  a  variable  indicating 

no  human  or  machine  translation  needed  for  a  system  pair  (+1),  machine 
translation  required  (0),  or  human  translation  required  (-1).  M  can  be 
augmented  by  multiplying  additional  matrices  (layers)  such  as  normalized 
bandwidth,  probability  of  connection  between  system  pairs,  mission 
capable  rate  for  systems,  normalized  cost,  system  reliability,  etc. 

n  n 

Step  3  -  Calculate  the  i-Score 

mij  ' 

i= 1  7=1 

n  n 

Step  4  -  Calculate  the  optimum  i-Score  I  t  =  V  V  mn  I  where 

P  <  J  J  J  I  tnopt 

i= 1  j= 1 
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Mopt  = 


CiiSii  ,  X 

lJ  lJ  \  s  jj  =max  j } 


is  the  maximally  upgraded  interoperability 


matrix  (i.e.,  upgrade  all  spins  that  can  be  upgraded  in  light  of  physical, 
fiscal  and  operational  constraints). 


Step  5  -  Calculate  the  Interoperability  Gap  /  =  / 


opt 


Step  6  -  Perform  interoperability  analysis  to  1)  determine  ways  of  closing 
the  interoperability  gap  through  spin  upgrades  or  using  common  systems, 

2)  determine  average  interoperability  spin,  3)  compare  operational  threads 
through  a  normalized  i-Score,  or  4)  visualize  the  interoperability  of  a 
thread  by  graphing  it  on  an  Interoperability  Terrain  graph,  (pp.  15-16) 

During  a  MCTSSA  MC3  SoS  assessment  in  2008,  in  addition  to  the  risk-based 
reporting  methodology  used  to  quantify  the  performance  of  the  C4  SoS,  a  demonstration 
of  the  i-Score  methodology  was  conducted  using  an  Immediate  Close  Air  Support  (ICAS) 
mission-based  test  thread  as  the  basis  for  the  demonstration.  The  ICAS  test  thread 
(Figure  10)  was  simplified  for  this  demonstration  to  ensure  all  activities  during  the 
execution  of  the  mission  thread  were  conducted  in  a  serial  manner  since  the  i-Score 
methodology  did  not  easily  accommodate  parallel  processes  (Sjoberg,  2008). 


Figure  1 0.  MC3  08  Modified  ICAS  Test  Thread. 

Modified  Immediate  Close  Air  Support  (ICAS)  mission-based  test  thread  used  to 
demonstrate  i-Score  methodology  during  MCTSSA  MC3  C4  SoS  assessment  in  2008 

(After:  Sjoberg,  2008). 
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The  systems  supporting  the  thread  activity  were  identified  as  the  Forward  Air 
Controller  (FAC),  denoted  as  System  1  (si);  the  Fire  Support  Coordination  Center 
(FSCC),  denoted  as  System  2  (S2);  the  Direct  Air  Support  Center,  denoted  as  System  3 
(S3)  and  the  Aircraft  (AC),  denoted  as  System  4  (S4).  With  the  test  thread  defined,  the  six 
step  i-Score  methodology  was  applied  as  follows: 

Step  1.  From  the  mission  thread  in  Figure  10,  the  ordered  set  T  of  systems 
supporting  the  ICAS  mission  thread  was  determined  as  reflected  in  Equation  3.1: 

T-{Sl,S3,S2,S3,S4,Si,S4,Si  S4,S3}  .  (3.1) 

Step  2.  The  interoperability  matrix  (4x4  matrix:  n=4  since  we  have  4  systems 
supporting  the  thread)  is  determined  by  Equation  3.2  (Ford  et  al.,  2006): 

M  =  [cijS  ,J  ,  (3.2) 

L  LJ  Mxn  x  ' 

where, 

C  =  Multiplicity  Matrix, 

S  =  Spin  Matrix. 

The  multiplicity  matrix  is  determined  by  taking  the  elements  of  T  two  at  a  time  in 
a  forward  direction  (direction  of  mission  thread  execution)  to  determine  set  A  (Ford  et  al., 
2006).  The  intent  is  that  this  methodology  accommodates  both  direct  interaction  as  data 
is  interchanged  between  components  and  indirect  interaction  that  accommodates  the 
upstream  influences  as  the  data  progresses  through  the  mission  thread.  For  the  ICAS 
thread  then  we  determine  set  A  in  Equation  3.3  as  follows: 

A  =  {(Si,S3),  (si,s2),  (si,s3),  (si,s4),  (si,Si),  (si,s4),  (s i ,S i),  (si,s4),  (si,s3), 

(S3,S2),  (S3,S3),  (S3,S4),  (S3, Si),  (S3,S4),  (S3, Si),  (S3,S4),  (S3, S3),  (S2,S3), 

(S2,S4),  (S2,Si),  (S2,S4),  (S2,Si),  (S2,S4),  (S2,S3),  (S3,S4),  (S3, Si),  (S3,S4), 

(53. 51) ,  (83,84),  (S3, S3),  (S4,Si),  (84,84),  (S4,Si),  (84,84),  (84,83),  (Si,S4), 

(51.51) ,  (Si,S4),  (Si,S3),  (S4,Si),  (S4,S4),  (S4,S3),  (Si,S4),  (s  1  ? S3) ?  (S45S3)}.  (3.3) 
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The  pair  (si,si)  appears  in  set  A  three  times  which  defines  C|  j,  and  pair  (si,S2) 
appears  in  this  set  one  time  which  defines  C\^  and  so  on  to  result  in  the  multiplicity 
matrix  defined  in  Equation  3.4  (Ford  et  al.,  2006): 


nxn 


3  15  6 

2  0  2  3 

4  13  6 

3  0  3  3 


(3.4) 


For  the  spin  matrix,  a  pair-wise  comparison  and  analysis  was  completed  for  each 
system  and  relative  interoperability  with  each  other.  The  results  of  the  analysis  are 
reflected  in  Table  2: 
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Spin 

element 

Spin 

Max 

Spin 

Rationale 

su 

1 

1 

All  systems  have  perfect  interoperability  with  themselves,  which 
is  to  say  sn-  =  Sji. 

SU 

0 

1 

The  FAC’s  Target  Location,  Designation  and  Hand-Off  System 
(TLDHS)  does  not  communicate  directly  to  the  FSCC’s 
Advanced  Field  Artillery  Tactical  Data  System  (AFATDS)  or 
Intelligence  Operations  Workstation  (IOW).  Instead  the  FAC 
sends  information  directly  to  the  DASC  which  forwards  the 
information  on  to  the  FSCC.  It  is  possible  for  the  TLDHS  to 
communicate  directly  with  the  FSCC  so  this  spin  is  upgradeable 
to  1. 

sn 

1 

1 

The  FAC’s  TLDHS  enables  direct  data  exchange  with  the  DASC. 

su 

1 

1 

The  FAC’s  TLDHS  enables  direct  data  exchange  with  the  aircraft. 

S2\ 

0 

1 

The  FSCC  must  pass  information  through  the  DASC  to 
communicate  to  the  FAC.  It  is  physically  possible  to  establish 
digital  communication  from  the  FSCC  to  the  FAC  directly  so  this 
spin  is  upgradeable  to  1 . 

S22 

1 

1 

The  FSCC  and  DASC  can  conduct  direct  data  exchange  using 
IOWs  and  AFATDS. 

S2A 

0 

0 

The  FSCC  cannot  conduct  direct  data  exchange  with  the  aircraft 
and  must  do  so  through  the  DASC.  This  is  not  upgradeable  since 
there  is  no  requirement  for  the  FSCC  to  interoperate  directly  with 
the  aircraft. 

% 

1 

1 

The  DASC  communicates  directly  with  the  FAC  using  IOW  or 
AFATDS  and  TLDHS. 

^32 

1 

1 

The  FSCC  and  DASC  can  interoperate  using  IOW  and  AFATDS. 

^34 

-1 

1 

The  DASC  only  exchanges  data  with  the  aircraft  using  a  human 
translator  over  voice  channels.  This  spin  is  upgradeable  since 
digital  communication  is  possible  between  these  two  entities. 

*41 

1 

1 

The  aircraft  can  interoperate  directly  with  the  FAC’s  TLDHS. 

^42 

0 

0 

The  FSCC  and  aircraft  cannot  directly  exchange  data  and  must  do 
so  through  the  DASC.  This  is  not  upgradeable  since  there  is  no 
requirement  for  the  FSCC  to  interoperate  directly  with  the 
aircraft. 

^43 

-1 

1 

The  aircraft  only  provides  data  to  the  DASC  using  a  human 
translator  over  voice  channels.  This  spin  is  upgradeable  since 
digital  communication  is  possible  between  these  two  entities. 

Table  2.  Spin  Analysis. 

Analysis  of  the  modified  ICAS  test  thread  results  in  assignment  of  Spin  values  for  each 
direct  interaction  between  components  in  the  test  thread  (From:  Sjoberg,  2008). 
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From  the  spin  analysis  a  spin  matrix  (S)  and  optimal  spin  matrix  (Sopt)  are 
determine  in  Equations  3.5  and  3.6  (Ford  et  al.,  2006): 


10  11 
0  110 
111-1 
10-11 


nxn 


1111 
1110 
1111 
10  11 


(3.5) 


(3.6) 


Now  with  the  multiplicity  matrix  (C),  spin  matrix  ( S)  and  optimal  spin  matrix 
(Sopl)  defined,  the  interoperability  matrix  (M)  and  optimal  interoperability  matrix  (Mopl  ) 
can  be  determined  as  reflected  in  Equations  3.7  and  3.8: 
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M 

4 
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(3.7) 


'3  15  6' 

2  0  2  3 

"1111" 

1110 

'3  15  6" 

2  0  2  0 

C  = 

4  13  6 

_3  0  3  3_ 

II 

to 

1111 

_!  0  1  1 

Mopt 

4  13  6 

_3  0  3  3_ 

(3.8) 


Step  3.  From  M,  an  i-Score  (1)  as  determined  from  Equation  3.9  (Ford  et  al., 

2006): 
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(3.9) 


n  n 

1 =Z2X  > 

<•=1  j= i 

is  then  calculated  for  the  ICAS  test  thread  as  1=2 1 . 

Step  4.  From  Mopt ,  the  optimum  i-Score  (Iopt)  as  determined  from  Equation  3.10 
(Ford  et  ah,  2006): 


I 


opt 


ZZ 

i= 1  j= 1 


(3.10) 


is  then  calculated  as  Iopt  =  42. 

Step  5.  The  Interoperability  Gap  (Igap)  as  determined  from  Equation  3.11  (Ford  et 
al.,  2006) : 

(311) 


is  then  calculated  with  Igap=  21 . 

Step  6.  From  the  initial  analysis,  two  interoperability  factors  were  identified  that 
could  close  the  interoperability  gap:  the  inability  of  the  DASC  to  digitally  communicate 
with  the  aircraft  and  the  inability  of  the  FSCC  and  FAC  to  directly  exchange  date. 
Addressing  the  first  factor  would  close  the  Igap  score  by  18  points.  Addressing  the  second 
factor  was  determined  to  have  little  operational  relevancy  (Sjoberg,  2008). 

To  provide  a  higher  level  of  fidelity  to  the  model,  an  improved  version  was 
offered  in  2008.  The  improved  i-Score  methodology  replaces  the  discrete  values  of  the 
spin  matrix  with  continuous  values  that  describe  each  system’s  relative  interoperability  in 
what  is  then  termed  a  resemblance  matrix.  The  modification  to  the  model  is  described  by 
Ford,  Colombi,  Graham  and  Jacques  (2008)  as  follows: 

The  nature  of  each  system  is  described  as  a  set  of  system  character  states, 
which  set  is  called  a  system  instantiation.  The  resemblance  of  each 
instantiation  pair  is  measured  using  a  distance  metric,  and  the  resulting  set 
of  resemblance  coefficients  is  given  as  a  resemblance  matrix.  The  i-Score 
interoperability  measurement  method  is  then  upgraded  by  replacing  the 
original  spin  matrix  with  the  resemblance  matrix.  Because  the  coefficients 
in  the  resemblance  matrix  represent  exact  measures  of  similarity  between 
systems,  based  upon  system  characters  pertinent  to  interoperability,  the 
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measure  of  interoperability  obtained  enjoys  more  fidelity  and  accuracy 
than  that  possible  with  the  original  spin-based  method,  (p.  2) 

As  applied  to  the  MAGTF  C4  SoS  assessment  effort,  i-Score  provides  a 
methodology  for  determining  and  quantifying  the  relative  interoperability  of  a  mission- 
based  test  thread  under  the  supposition  that  the  optimal  interoperability  of  any  SoS  is 
obtained  with  maximum  direct  digital  communications  between  components  (no  human 
or  machine  translation  required).  The  maturity  of  the  interface  between  systems  or 
components  in  the  SoS  serves  as  the  basis  for  the  SoS  assessment  then,  with  direct  digital 
communications  representing  the  highest  maturity  and  human  translation  representing  the 
lowest  maturity.  From  an  operational  thread  perspective,  there  are  often  many  ways  to 
execute  a  mission  or  task  with  varying  degrees  of  human  interaction  (multiple 
combinations  of  systems  and  procedures  available  to  accomplish  the  same  mission). 
Subsequently,  a  test  organization  must  select  from  a  number  of  possible  test  threads  that 
meet  the  mission  requirement.  In  this  respect,  the  i-Score  methodology  can  assist  with 
defining  the  test  threads  that  promise  optimal  interoperability  (threads  with  least  human 
translation  required)  and  help  narrow  the  selection  for  inclusion  in  a  test  event. 
Additionally,  during  execution  of  the  test  thread  for  an  assessment  event,  the  relative 
impact  of  a  deviation  during  thread  execution  (e.g.,  human  translation  required  due  to 
component  translation  failure)  could  be  immediately  quantified  through  a  recalculation  of 
the  i-Score.  Further  analysis  of  the  improved  i-Score  methodology  may  yield  further 
application  with  added  value  to  a  C4  SoS  assessment  effort. 

2.  Interoperability  Quotient 

The  Interoperability  Quotient  (IQ)  process  was  developed  by  Northrop  Grumman 
Corporation  to  address  complex  SoS  testing  related  to  the  Navy’s  CVN-21  next 
generation  Aircraft  Carrier  program.  IQ  was  presented  to  the  Joint  test  community 
during  the  June  2007  JMETC  User  Group  Conference  in  Dulles  VA,  as  a  more  objective 
approach  to  performance  assessment  during  Joint  SoS  test  events.  Much  like  i-Score,  this 
methodology  addresses  the  lack  of  a  quantitative  measure  for  assessing  C4  SoS 
interoperability:  noting  that  interoperability  compliance  with  measures  of  performance 
such  as  NR-KPP  lack  objective  metrics  (Lawver,  2007). 
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The  test  process  looks  at  interoperability  from  two  perspectives:  first,  from  the 
perspective  of  data  transfers  between  individual  components  of  the  SoS  (external  tests) 
and  second,  from  the  perspective  of  data  processes  within  the  individual  components  of 
the  SoS  (internal  tests).  This  approach  is  referred  to  by  Northrop  Grumman  as  the 
“interoperability  test  stack”  or  “IQ  Stack”  to  draw  parallels  to  modular  functional  data 
models  like  the  Open  System  Interconnection  (OSI)  7-layer  stack  (Lawver,  2007). 

As  depicted  in  Figure  11,  the  external  tests  look  at  data  interaction  between  the 
systems  within  the  SoS  in  terms  of  vulnerability,  Web  standards  compliance,  edge 
condition  testing,  Service  Oriented  Architecture  (SOA)  technologies,  data  transmission 
bandwidth,  corrupted  data  and  security.  The  internal  tests  look  at  attributes  of  the 
individual  systems  within  the  SoS  in  terms  of  Human  Computer  Interface  (HCI)  and 
Symbology,  business  rules  for  data  fusion,  Information  Assurance  (IA)  and  semantic 
interoperability  (Lawver,  2007). 


E.g.,  TCP/IP,  File  Store, 

Screen  Display  11|Q  g^,, 


Component’s  “Out”  Data  Access 


Component’s  Boundary 
(External  Tests) 


Component’s  Internal  Logic 
(Internal  Tests) 


Component’s  Boundary 
(External  Tests) 


Component’s  “In”  Data  Access 


E.g.,  TCP/IP,  File  Store 


HCI  and  Symbology 


Business  Rules  for  Fusion 


Information  Assurance 


Context-level  Semantics 


MSG  -level  Semantics 


Field-level  semantics 


•  •  • 


^Much  activity  here 


JITC  &  NCTSI  Test 
these  for  TADILs 


The  straight-forward  test  issues  are  generally 
being  addressed,  but  most  of  the  hard/complex 
challenges  remain.  These  are  the  focus  of  IQ. 


Vulnerability 


Web  Standards  compliance 
(e.g.,  XML) 


Edge  Conditions  Testing 


SOA  Technologies 
(e.g.,  SOAP) 


Data  transmission  bandwidth 


Corrupted  data  sets 


Security 


j-Some  activity  here 
j-  Much  activity 


here 


j-Much  activity  here 
j-Much  activity  here 


(Note:  examples  are  not  hierarchical) 


Figure  11.  Northrop  Grumman  “IQ  Stack”. 

The  Northrop  Grumman  IQ  Stack  depicts  aspects  of  both  internal  and  external  data 
interactions  between  components  in  a  SoS  (From:  Lawver,  2007). 
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The  IQ  process  is  conducted  within  the  context  of  an  operational  mission  test 
thread  that  provides  the  basis  for  evaluation  of  the  components  (command  centers, 
platforms,  individual  systems  or  applications)  in  the  SoS.  As  the  thread  is  executed, 
internal  and  external  test  artifacts  are  collected  for  the  components  and  manually  scored 
based  on  an  established  scoring  criteria  (numerical  score  associated  with  each  possible 
outcome).  As  the  numeric  scores  are  determined,  a  color  code  (green-pass,  yellow- 
caution,  and  red-fail)  is  also  assigned.  Both  internal  and  external  test  scores  are  rolled  up 
for  an  overall  score  and  then  those  scores  are  rolled  up  to  provide  an  overall  component 
IQ  score  (maximum  200-point  scale)  and  overall  color  code  for  the  component  (Lawver, 
2007).  This  roll  up  methodology  as  depicted  in  Figure  12,  also  allows  for  specific 
weighting  of  individual  tests  to  accommodate  specific  design  capabilities  to  help 
normalize  the  score  to  the  200-point  scale  (Lawver,  2007).  Of  note  is  that  the  validity  of 
the  roll-up  methodology  is  dependent  on  the  scoring  methodology  selected  for  the 
internal  and  external  tests.  If  scoring  is  not  based  on  a  ratio  scale,  the  summation  and  any 
statistical  inferences  will  not  be  valid. 


Figure  12.  IQ  Computation  Roll  Up  Methodology. 

IQ  rollup  methodology  is  depicted  for  the  GCSS-M  component  in  the  CVN-21  SoS 

(From:  Lawver,  2007). 
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To  help  better  visualize  and  report  the  results  of  an  interoperability  assessment, 
Northrop  Grumman  also  developed  a  companion  Web  portal  tool  called  “Auto  IQ” 
(Lawver,  2007)  that  provides  a  visual  interpretation  of  the  data  and  allows  drill  down  to 
see  how  the  individual  scores  are  rolled  up  for  the  overall  IQ  component  score.  Depicted 
in  Figure  13,  the  intent  is  to  provide  a  means  for  analysts  to  drill  down  into  the  data  for  a 
better  understanding  of  the  overall  component  score  and  to  provide  the  component 
program  manager  with  insight  into  the  individual  test  attributes  that  may  serve  as  means 
to  identify  areas  that  can  be  addressed  to  improve  component  interoperability. 


Figure  13.  “Auto  IQ”  Web  Portal. 

The  Auto  IQ  Web  Portal  provides  a  convenient  method  to  display  SoS  assessment  results 
and  a  drill  down  capability  to  see  how  the  components  of  a  SoS  contributed  to  the  overall 

IQ  score  (From:  Lawver,  2007). 


From  a  MAGTF  C4  SoS  assessment  perspective,  the  IQ  methodology  provides  a 
similar  approach  to  the  MC3  risk-based  reporting  methodology  with  performance  data 
obtained  from  the  individual  systems  during  execution  of  the  test  thread  used  to  develop  a 
measure  of  risk  associated  with  the  success  of  executing  a  test  thread  in  the  SoS. 
However,  the  IQ  test  model  examines  both  internal  and  external  interfaces  through 
performance  metrics  that  once  defined,  are  intended  to  provide  a  quantifiable  and 
repeatable  metric  that  characterizes  the  overall  interoperability  of  the  SoS. 
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3.  Dynamic  Software  Architecture  Visualization  and  Evaluation 
(DynSAVE) 

While  not  specifically  offering  a  methodology  that  leads  to  an  interoperability 
score,  the  Fraunhofer  Center  for  Experimental  Software  Engineering  Maryland  (FC-MD) 
provides  an  approach  for  assessing  the  key  interoperability  aspects  of  digital  information 
exchange.  The  FC-MD  is  a  not  for  profit,  applied  research  organization  associated  with 
University  of  Maryland  that  examines  advanced  software  engineering  techniques  and 
conducts  “applied  research  in  software  architecture,  verification  &  validation,  process 
improvement  and  measurement”  (Ganesan,  Lindvall,  Bartholomew,  Blau,  McComas,  & 
Cammarata,  2008,  p.  4).  To  address  the  issues  with  performance  of  complex  software 
SoS,  the  FC-MD  developed  an  approach  for  assessing  software  SoS  performance  and 
identifying  problems  with  communications  between  systems.  The  approach  was  derived 
from  an  initial  effort  in  support  of  Johns  Hopkins  University  Applied  Physics  Laboratory 
(JHU/APL)  Space  Department  and  the  development  of  Mission  Operations  Center 
(MOC)  system  software  used  by  NASA  missions.  The  MOC  system  software  used  a 
shared  architecture  called  the  Common  Ground  System  (CGS)  and  because  of  its  age  (10 
years  old),  presented  challenges  when  MOC  software  modification  were  required  in 
support  of  new  mission  requirements  (Stratton,  Sibol,  Lindvall,  &  Costa,  2006). 

To  address  this  challenge,  FC-MD  in  a  collaborative  effort  with  the  Fraunhofer 
Institute  for  Experimental  Software  Engineering  (IESE)  developed  and  applied  the 
Software  Architecture  Visualization  and  Evaluation  (SAVE)  tool  and  process: 

This  process  comprised  defining  a  planned  architecture  including 
architectural  goals  and  design  rationale,  generating  a  high-level 
description  of  the  actual  architecture  from  the  legacy  software,  identifying 
deviations  between  the  planned  and  actual  architecture,  creating  a  new 
target  architecture,  and  creating  a  roadmap  to  align  on-going  systems 
development  and  maintenance  with  new  target  architecture.  (Stratton  et 
al.,  2006,  p.  1) 

The  SAVE  tool  provides  a  means  to  automatically  extract  architectural  views 
from  the  application’s  source  code  and  checks  the  compliance  of  source  code  with  the 
planned  architecture  (Ganesan  et  al.,  2008)  as  depicted  in  Figure  14. 
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Figure  14.  SAVE  Tool  Analysis. 

The  SAVE  tool,  provides  a  means  to  automatically  extract  architectural  views  from  the 
application’s  source  code  for  comparison  with  the  planned  architect  (From:  Ganesan  et 

al„  2008,  p.  9). 


Primarily  applicable  to  single  software  applications,  SAVE  conducts  static 
analysis  of  software  systems  written  in  a  variety  of  languages  (C/C++,  Java,  Delphi, 
Simulink,  and  others)  to  identify  variances  (violations  of  interaction  between 
components)  in  planned  versus  implemented  software  architecture  (Stratton  et  al.,  2006). 

In  a  follow-on  effort,  FC-MD  extended  the  functionality  of  SAVE  to  provide  an 
ability  to  conduct  dynamic  analysis  of  a  software  SoS.  The  new  product  called 
DynSAVE  (Dynamic  SAVE)  provides  both  structural  and  behavioral  architecture 
analysis  of  a  SoS.  Asserting  that  the  reliability  of  inter-systems  communication 
determines  the  success  of  the  overall  capability  of  a  SoS,  DynSAVE  provides  an 
approach  for  capturing,  processing,  and  visually  representing  communications  behavior 
of  a  SoS  for  analysis  (Stratton,  Sibol,  Lindvall,  Ackermann  &  Godfrey,  2009). 


48 


The  challenge  associated  with  inter-systems  communications  typically  stem  from 
individual  systems  implementation  of  communications  protocols.  While  the  protocols 
are  typically  defined  in  the  system’s  Interface  Control  Document  (ICD),  a  number  of 
factors  influence  the  implementation  of  protocols  that  can  result  in  degraded  SoS 
performance: 

1 .  The  systems  are  often  developed  independently  from  each  other  by 
different  development  teams  at  different  locations. 

2.  The  communication  behavior  of  each  individual  system  is  not  and 
cannot  be  fully  tested  in  the  environment  it  will  eventually  operate 
in,  but  rather  by  using  components  that  simulate  the 
communication  of  other  systems. 

3.  The  ICD  that  specifies  the  protocol  is  ambiguous  as  it  omits  details 
allowing  developers  to  interpret  the  protocol  differently. 

4.  The  ICD  consists  of  hundreds  of  pages  of  text  written  in  natural 
language,  making  it  difficult  for  developers  to  fully  understand  and 
implement.  In  the  case  of  CFDP,  most  implementations  only 
support  the  commonly  used  features  of  the  protocol,  so  issues  with 
integration  of  differing  subset  implementations  emerge. 

5.  Many  clients  exist  that  implement  the  ICD  protocol  and  it  would 
be  a  significant  effort  to  update  them  if  the  protocol  was  changed. 

6.  Violations  of  the  protocol  are  not  clearly  visible  but  manifest 
themselves  as  some  kind  of  misbehavior.  (Stratton  et  al.,  2009,  p. 

3) 

Subtle  difference  in  protocol  implementation  can  result  in  anomalies  in 
communications  behavior  that  impact  reliability  and  efficiency  of  the  SoS.  The 
DynSAVE  methodology  classifies  the  anomalies  in  terms  of  Sequence  (defining  the  order 
of  interaction  between  systems  during  message  exchange),  Parameters  (control  signals 
and  data  within  a  message  that  controls  system  behavior)  and  Timing  (time  constraints  of 
message  exchange  between  systems).  The  DynSAVE  approach  depicted  in  Figure  15 
involves  capturing  raw  data  during  communication  exchanges,  mapping  the  raw  data  to 
protocol,  and  then  visualizing  the  communications  behavior  in  a  sequence  diagram  for 
interpretation  and  evaluation  of  the  message  exchange  between  systems  within  a  SoS. 
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Figure  15.  DynSAVE  Process. 

The  DynSAVE  Process  captures  raw  data  for  interpretation  and  visualization  in  a 
sequence  diagram  (After:  Stratton  et  al.,  2009,  p.  6). 


While  the  MC3  MAGTF  C4  SoS  assessment  approach  has  looked  at  capturing 
emergent  or  unexpected  behavior  within  the  SoS  during  execution  of  a  SoS  performance 
assessment  in  a  qualitative  manner,  DynSAVE  may  provide  a  means  for  applying  a  more 
quantitative  measure  to  anomalous  behavior.  Although  limited  to  IP-based  protocols, 
DynSAVE  can  provide  greater  insight  into  the  origins  of  anomalous  behaviors  in  a 
complex  SoS  in  terms  of  the  sequence,  parameters  and  timing  attributes  of  digital 
communications:  offering  a  means  of  quantifying  the  anomalous  behavior  and  related 
measure  of  efficiency  and  reliability  of  a  SoS  through  those  attributes. 
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c. 


SUMMARY 


A  key  and  certainly  foundational  aspect  of  the  C4  SoS  is  the  myriad  of  software 
systems  that  process  the  data  necessary  to  provide  the  C2  capability.  The  interoperability 
assessment  of  both  i-Score  and  IQ  address  this  with  a  specific  focus  on  interfaces  and 
data  exchange  between  components  that  are  primarily  software  systems  or  software 
systems  within  a  functional  component  of  the  SoS  (e.g.  AFATDS  in  the  DASC 
component).  As  depicted  in  Table  3,  the  i-Score  methodology  examines  the  interface 
from  a  macroscopic  level  assigning  a  value  to  the  relative  maturity  level  of  the  digital 
interface  between  components.  The  IQ  methodology  goes  a  step  further  suggesting  that 
both  interface  between  components  and  internal  processing  within  a  component  are 
necessary  to  derive  an  overall  SoS  interoperability  metric.  However  examining  the 
internal  logic  and  boundaries  of  the  components  within  a  complex  SoS  is  not  trivial.  To 
address  this  complexity,  the  FC-MD  DynSAVE  tool  and  process  provides  an  approach 
for  identifying  and  visualizing  digital  communications’  anomalies  to  convey  a  better 
understanding  of  the  information  exchange  attributes  that  influence  both  the  efficiency 
and  reliability  aspects  of  a  SoS. 
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Assessment 

Attributes  and  Applicability  to 

Limitations 

Tool 

MAGTF  C4  SoS  Assessment 

i-Score 

•  Quantitative  approach  for  measuring 

• 

Accommodation  of  parallel 

SoS  interoperability  through  metric 

processes  in  test  thread 

associated  with  interfaces  between 

• 

No  direct  correlation  between 

components 

interoperability  score  and  SoS 

•  May  provide  means  for  selecting  test 

performance  metrics 

thread  based  on  i-Score  metric  and 

relative  interoperability 

•  May  provide  a  means  to  quantify 

deviation  from  intended  thread  execution 

during  an  assessment 

IQ 

•  Quantitative  approach  for  measuring  SoS 

• 

Interoperability  scoring 

interoperability  through  metric 

methodology  relies  heavily  on 

associated  with  interfaces  between 

subjective  scoring  and 

components  and  processing  within  a 

weighting  of  interoperability 

component. 

attributes 

•  May  provide  a  methodology  for  more 

• 

Summation  methodology  only 

clearly  defining  SoS  assessment  metrics 

valid  if  scoring  is  based  on  a 

•  May  provide  a  means  to  better  visualize 

ratio  scale 

how  lower  level  metrics  contribute  to 

• 

No  direct  correlation  between 

overall  SoS  assessment  metric 

interoperability  score  and  SoS 

performance  metrics 

DynSAVE 

•  Tool  and  process  for  identifying  and 

• 

Applicable  to  software  C4  SoS 

visualizing  anomalies  associated  with 

transactions  only  (does  not 

digital  communications  within  a  SoS 

accommodate  non-digital 

•  May  provide  a  means  to  quantify  SoS 

anomalies  and  aspects  of  efficiency  and 

reliability 

transactions  and  components 

that  may  contribute  to  the 

efficiency  and  reliability  of  the 

SoS  as  a  whole) 

Table  3.  Assessment  Tool  Summary. 

Attributes,  application  and  limitations  of  three  SoS  assessment  tools. 
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While  the  tools  described  in  this  section  all  offer  unique  methodologies  for 
assessing  SoS  performance  that  may  help  to  better  quantify  the  performance  of  a 
MAGTF  C4  SoS,  they  are  also  very  similar  in  that  they  each  examine  information 
exchanges  at  the  component  level  to  derive  an  overall  SoS  performance  assessment 
measure  in  terms  of  interoperability.  In  the  next  section,  multicriteria  identification  is 
examined  as  a  means  to  accommodate  not  only  the  interoperability  measure  of  a  SoS  but 
other  attributes  as  well  for  a  more  holistic  and  complete  assessment  of  the  SoS. 
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IV.  IMPROVING  THE  SOS  MODEL  THROUGH 
MULTICRITERIA  IDENTIFICATION 


A.  INTRODUCTION 

The  JMETC  Joint  SoS  testing  approach  and  MCTSSA  C4  SoS  assessment 
approach  share  similar  features.  For  both,  execution  of  a  mission-based  test  thread 
provides  the  stimulus  for  the  components  of  the  SoS  and  serves  a  basis  for  providing  an 
indication  of  SoS  suitability  (ability  to  complete  the  test  thread).  For  both,  the  mission- 
based  test  thread  is  executed  in  a  SoS  that  is  to  varying  degrees,  an  abstract  model  of  the 
actual  SoS  employed  by  the  end-user  (LVC  concept).  The  use  of  Modeling  and 
Simulation  (M&S)  is  not  only  necessary  from  a  practical  standpoint,  but  also  aligned  with 
the  guidance  provided  in  the  2008  Systems  Engineering  Guide  for  Systems  of  Systems : 

Models,  when  implemented  in  an  integrated  analytical  framework,  can  be 
an  effective  means  of  understanding  the  complex  and  emergent  behavior 
of  systems  that  interact  with  each  other.  They  can  provide  an  environment 
to  help  the  SoS  SE  team  to  create  a  new  capability  from  existing  systems 
and  consider  integration  issues  that  can  have  a  direct  effect  on  the 
operational  user. 

Because  it  can  be  difficult  or  infeasible  to  completely  test  and  evaluate 
capabilities  of  the  SoS,  M&S  can  be  very  effectively  applied  to  support 
test  and  evaluation  at  different  stages  throughout  the  SoS  SE  process.  In 
particular  the  SoS  SE  team  should  consider  M&S  of  the  SoS  to  understand 
the  end-to-end  performance  of  the  overall  SoS  prior  to  implementation.  In 
some  cases  it  is  advisable  for  the  SoS  SE  team  to  adopt  a  model-based 
process.  (OSD,  2008,  p.  10) 

In  another  level  of  abstraction  beyond  the  physical  models  that  are  instantiated  as 
the  basis  of  SoS  testing  and  assessment  by  JMETC  and  MCTSSA,  the  construction  and 
use  of  mathematical  models  that  describe  the  behavior  of  a  SoS  would  also  seem 
relevant.  A  mathematical  model  would  provide  a  means  for  assessing  behavioral 
characteristics  resulting  from  changes  to  key  SoS  attributes  in  pursuit  of  understanding 
how  and  where  improvements  to  the  actual  SoS  can  be  made.  However,  developing  a 
mathematical  model  that  represents  the  complex  behavior  of  a  SoS  is  a  significant 
challenge.  Furthermore,  even  with  that  challenge  addressed,  obtaining  an  understanding 
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of  how  adequate  or  how  closely  the  model  can  predict  actual  behavior  is  necessary  before 
that  model  can  be  extended  and  used  for  any  decision  analysis. 

A  set  of  mathematical  equations  that  describe  the  operation  or  performance  of  a 
system  is  fundamental  to  optimization  problems.  The  equation  variables  and  constraints 
associated  with  the  equations  define  the  design  space  and  serve  as  the  basis  of  analysis  in 
pursuit  of  optimizing  performance  criteria.  Looking  at  this  process  in  reverse  to 
determine  and  improve  a  mathematical  model  from  observed  operation  and  performance 
of  a  system  is  the  basis  of  the  identification  process.  Multicriteria  identification  provides 
an  approach  for  determining  the  adequacy  of  a  mathematical  model  relative  to  the  actual 
behavior  of  the  system  or  SoS  that  the  model  represents.  Once  the  mathematical  model  is 
determined  adequate,  it  can  then  be  used  for  improving  physical  models  or  prototypes.  In 
discussion  of  establishing  adequacy  through  various  identification  methods,  Statnikov 
and  Matusov  (2002)  offer  this  perspective: 

In  the  most  common  usage,  the  term  “identification”  means  construction 
of  the  mathematical  model  of  a  system  and  determination  of  the 
parameters  (design  variables)  of  the  model  by  using  the  information  about 
the  system  response  to  known  external  disturbances.  In  a  sense, 
identification  problems  are  inverse  with  respect  to  optimization  problems. 

(p.  88) 

From  a  SoS  perspective,  the  mathematical  model  would  ideally  represent  performance  at 
the  SoS  level  with  determination  of  the  parameters  of  the  model  obtained  during 
execution  of  a  mission  task  (external  disturbance).  The  feasibility  of  constructing  a 
mathematical  model  to  represent  the  MAGTF  C4  SoS  and  conceptual  use  of  multicriteria 
identification  for  establishing  and  improving  the  test  environment  as  well  as  adapting 
elements  of  the  multicriteria  identification  process  as  part  of  the  SoS  assessment  process 
itself  are  examined  below. 

B.  MAGTF  C4  SYSTEM  OF  SYSTEMS  PROTOTYPE 

The  representative  MAGTF  C4  SoS  that  serves  as  the  foundation  of  the 
assessment  process  consists  of  computers,  databases  and  applications  that  receive, 
manipulate,  send  and  display  data  over  a  communications  and  computer  network 
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infrastructure  for  use  by  machines  or  humans  during  the  prosecution  of  a  mission  thread. 
While  often  consisting  of  many  of  the  same  components  found  in  the  Marine  Corps 
Operating  Force’s  C4  SoS,  the  representative  C4  SoS  in  the  test  environment  is  usually 
an  abstraction  of  the  actual  environment. 

The  test  environment  is  a  physical  model  of  the  actual  SoS  and  may  also  be 
considered  a  prototype  that  can  serve  as  an  environment  for  testing  new  C2  applications, 
modifying  system  configurations  or  modifying  communications  protocols  or  networking 
attributes  that  we  hope  will  improve,  or  at  least  do  no  harm  to  the  actual  C4  SoS.  As  we 
modify  the  prototype,  the  hope  is  the  resulting  performance,  interoperability  measure  or 
other  attributes  of  the  SoS  will  adequately  reflect  what  we  would  expect  in  the  actual 
SoS.  A  complementary  mathematical  model  that  describes  the  behavior  of  the  SoS 
would  provide  a  means  to  improve  the  physical  model  or  prototype  through  the 
multicriteria  identification  process. 

C.  MAGTF  C4  SYSTEM  OF  SYSTEMS  MATHEMATICAL  MODEL 

Mathematical  modeling  of  C2  communications  and  network  architectures  is 
pursued  for  a  variety  of  uses.  One  of  the  most  widely  known  modeling  constructs  used 
within  the  Department  of  Defense  (DoD)  is  OPNET  Modeler.  This  modeling  construct 
focuses  primarily  on  communications  systems  and  provides  a  capability  for 
communications  systems  planning  and  performance  prediction.  Wargaming  simulations 
like  JANUS(T)  use  mathematical  models  that  serve  as  the  foundation  for  determining 
predictive  outcome  during  military  operational  scenarios.  These  models  focus  on 
performance  of  military  units  with  inherent  C2  capabilities  that  can  infer  C2  performance 
(Ingber,  Fujio,  &  Wehner,  2001). 

Developing  a  mathematical  model  that  describes  the  behavior  of  the  MAGTF  C4 
SoS  hints  at  the  very  essence  of  the  SoS  assessment  challenge:  defining  the  key  SoS 
performance  criteria.  The  key  performance  criteria  that  serve  as  the  basis  of  an 
assessment  are  central  to  the  construction  of  a  mathematical  model  that  describes  the 
SoS.  The  i-Score  and  IQ  methodologies  quantify  the  SoS  by  applying  the  principles  of 
decomposition  and  aggregation  of  large-scale  systems  with  an  analysis  and  a  scoring 
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construct  that  defines  an  interoperability  measure  as  the  key  performance  criteria. 
Measuring  interoperability  at  the  individual  system  level  is  an  approach  shared  by 
i-Score,  IQ  and  DynSAVE.  Improving  the  interoperability  between  components  through 
i-Score,  IQ  or  DynSAVE  analysis  would  imply  improved  efficiency  (reduction  in  human 
or  machine  processing  requirements  within  the  SoS  as  the  thread  is  executed)  and 
improved  effectiveness  (reduction  in  processing  implies  potential  for  quicker  execution 
of  the  thread  with  less  potential  for  introduction  of  error  during  translation  from  one 
component  to  the  next).  These  approaches  each  provide  a  means  for  defining  and 
measuring  attributes  at  the  systems  level  for  aggregation  into  a  meaningful  overall 
interoperability  measure  that  should  correlate  to  the  overall  efficiency  and  effectiveness 
of  the  SoS. 

While  IQ  and  DynSave  provide  a  means  of  deriving  a  measure  of  interoperability 
through  observed  and  quantitative  assessment  of  a  SoS,  only  the  i-Score  methodology 
provides  an  approach  to  describing  a  SoS  in  terms  of  a  mathematical  model.  The  i-Score 
approach  to  modeling  a  SoS  in  terms  of  interoperability  is  not  unique.  Other  approaches 
have  been  proposed  to  include  a  model  that  describes  the  SoS  in  terms  of  interoperability 
and  complexity  presented  in  A  Model  for  Assessing  the  Performance  of  Interoperable, 
Complex  Systems  by  Thomas  Huynh  and  John  Osmundson  (2006).  However,  the  use  of  a 
mathematical  representation  of  a  SoS  in  terms  of  interoperability  is  limited  unless  we  can 
somehow  correlate  the  interoperability  measure  with  measurable  performance  criteria  of 
the  SoS.  To  address  this  common  challenge,  prototype  experimentation  may  provide  a 
means  to  help  determine  the  correlation  between  an  interoperability  measure  and 
performance  of  the  SoS  and  then  help  improve  the  model  through  parametric 
identification  (improving  or  determining  the  appropriate  numerical  values  of  the  equation 
coefficients). 

1.  Prototype  Experimentation  Data  to  Develop  and  Improve  Model 

In  a  broader  perspective,  we  would  like  to  characterize  the  C4  SoS  in  terms  of 
overall  C2  effectiveness  (ability  to  efficiently  and  effectively  provide  a  C2  capability)  as 
the  main  criterion.  The  SoS  should  provide  a  C2  capability  while  consuming  as  few 
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resources  as  possible  (measure  of  efficiency)  while  enabling  the  successful  execution  of  a 
mission  thread  as  quickly  as  possible  (measure  of  effectiveness).  The  interoperability 
measure  addresses  this  in  part,  but  with  focus  solely  on  thread  execution  and 
interoperability  between  systems,  other  attributes  of  the  C4  SoS  that  are  necessary  to 
provide  indirect  or  background  C2  activities  may  not  be  adequately  accommodated. 
Activities  such  as  position  reporting,  DISN  services,  and  maintaining  Situational 
Awareness  (SA)  may  contribute  directly  to  the  execution  of  a  mission  thread,  but  are  also 
critical  background  processes  that  enable  decision  making  to  ensure  the  appropriate 
mission  is  executed  at  the  appropriate  time  for  successful  execution  of  a  warfighting 
campaign.  From  that  perspective,  a  mathematical  model  would  need  to  accommodate 
those  activities  as  variables  within  the  model  (demand  for  services  varies  with  the 
operational  tempo  and  battle-rhythm  that  is  driven  by  the  operational  scenario  and 
echelon).  Those  variables  as  well  as  the  activities  directly  engaged  with  execution  of  the 
mission  thread  are  consumers  of  bandwidth  and  the  quality  of  service  related  to  those 
activities  is  dependent  on  bandwidth  availability.  Availability  of  bandwidth  is  generally 
a  fixed  constraint  within  the  SoS  although  allocation  of  bandwidth  within  the  SoS  to 
specific  C2  services  can  be  optimized  to  best  meet  operational  needs.  Ideally,  although 
beyond  the  scope  of  this  study,  a  mathematical  interoperability  model  would  be 
correlated  with  measurable  attributes  of  C2  effectiveness  (such  as  bandwidth  efficiency, 
resource  usage  and  time  to  execute  mission  tasks),  while  accommodating  the  variable 
background  activities  and  fixed  bandwidth  constraint. 

a.  Simulation  and  Stimulation  Considerations 

To  accommodate  the  C2  background  activities  in  the  model,  some  thought 
must  be  given  to  how  they  can  be  represented  in  the  prototype.  The  operational  scenario 
serves  as  a  reference  for  the  mission-based  test  thread  and  drives  the  demand  for  C2 
services.  However,  the  demand  for  C2  background  activities  is  largely  dependent  on  the 
battle  rhythm  and  operational  tempo  during  mission  execution  and  not  necessarily 
dependent  on  mission  execution  itself.  Real-world  operational  data  can  provide  a  basis 
for  determining  the  variations  in  C2  background  activities  (Position  Location  Information 

or  track  load  and  DISN  services  such  as  VTC,  telephone,  and  collaboration  services)  that 
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can  be  replicated  in  the  C4  SoS  prototype  thorough  various  simulation  and  stimulation 
applications.  Executing  the  test  thread  in  the  prototype  environment  under  various  mixes 
and  load  of  C2  background  activities  will  induce  more  realistic  behavior  necessary  for 
developing  a  higher  fidelity  mathematical  model. 

The  use  of  virtualization  technology  may  also  offer  a  more  efficient  means 
to  establish  a  large-scale  C4  SoS  model.  MCTSSA  experimented  with  using 
virtualization  technology  to  replicate  a  MAGTF  C4  SoS  during  the  2008  MC3 
assessment  effort.  Virtualization  technology  provided  a  means  to  rapidly  construct  the 
C4  architecture  to  validate  individual  component  configuration  settings  and  test 
procedures  to  be  used  during  the  assessment.  Once  configuration  settings  and  test 
procedures  were  validated,  the  C4  architecture  was  constructed  with  actual  C4 
components  for  the  assessment.  This  resulted  in  considerable  time  savings  (2-3  days) 
over  previous  MC3  events  by  allowing  the  test  procedure  validation  effort  to  be 
conducted  in  parallel  with  other  pre-assessment  preparation  efforts.  Use  of  a  virtualized 
environment  to  conduct  the  actual  assessment  was  not  considered  due  to  the  unknown 
differences  in  performance  between  the  virtual  and  actual  C4  SoS  models  (Marine  Corps 
Tactical  Systems  Support  Activity,  2009). 

b.  Data  Collection  Considerations 

Performance  data  related  to  C2  effectiveness  can  be  obtained  from  a 
measure  of  the  time  to  execute  the  mission  thread.  Performance  data  related  to  C2 
efficiency  may  require  use  of  tools  to  measure  bandwidth  utilization  as  a  measure  of 
efficiency  or  new  tools  like  DynSAVE  that  capture  the  behavior  of  complex  SoS  data 
transactions  for  more  detailed  efficiency  analysis. 

In  an  operational  C4  SoS,  network  optimization  and  Quality  of  Service 
(QoS)  management  are  performed  through  use  of  commercial  software  tools  such  as 
Solar  Winds’  Orion  Network  Performance  Monitor  associated  with  the  Joint  Network 
Management  System  (JNMS)  program  of  record,  Cisco’s  suite  of  network  management 
products  associated  with  the  TDN  program  of  record,  and  Referenda’s  LiveAction 
network  management  solution  associated  with  an  Office  of  Naval  Research  (ONR) 
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development  effort.  While  these  tools  serve  as  an  integral  part  of  the  SoS,  they  are  not 
always  included  in  the  test/assessment  prototype  environment  since  they  do  not  directly 
contribute  to  the  execution  of  a  mission  thread.  However,  if  C2  background  activities  are 
incorporated  within  the  SoS  prototype,  these  tools  are  necessary  to  ensure  the  network  is 
configured  to  reflect  realistic  operational  requirements  with  respect  to  QoS  and  network 
performance  optimization.  Further,  while  these  tools  are  an  element  within  the  SoS,  one 
of  their  key  functions  is  to  provide  internal  diagnostics  relative  to  the  SoS  network 
performance  and  while  considered  intrusive  from  a  test  environment  data  collection 
perspective,  may  still  provide  critical  insight  into  and  measurement  of  performance 
efficiency  from  a  resource  utilization  perspective. 

2.  Multicriteria  Identification 

The  complexity  of  a  MAGTF  C4  SoS  may  prohibit  development  of  an  adequate 
mathematical  model  in  terms  of  a  single  performance  criteria.  Bandwidth  efficiency, 
resource  usage  and  time  to  execute  mission  tasks  are  three  criteria  identified  earlier,  but 
there  could  be  many  other  performance  criteria  that  must  be  considered  to  adequately 
describe  the  SoS — that  is  to  say,  a  mathematical  model  that  reflects  the  correlation 
between  the  interoperability  measure  and  bandwidth  efficiency  performance  measure 
may  differ  from  the  mathematical  model  that  reflects  the  correlation  between  the 
interoperability  measure  and  time  to  execute  a  mission  task.  Yet,  both  may  be  needed  to 
adequately  describe  the  SoS.  This  would  indicate  that  correlating  interoperability  with 
SoS  performance  may  very  well  be  a  multicriteria  problem.  Developing  and  improving 
the  mathematical  relationship  between  interoperability  and  SoS  performance  then  would 
involve  a  multicriteria  identification  process:  observing  performance  of  the  prototype 
during  mission-based  thread  execution  and  through  experimentation  develop  and  improve 
the  mathematical  relationship. 

a.  Use  of  Adequacy  Criteria  to  Improve  the  Models 

Adequacy  criteria  (or  proximity  criteria)  represent  the  discrepancy 
between  the  physical  prototype  model  and  the  mathematical  model  (Statnikov  & 
Matusov,  2002).  Once  a  mathematical  model  is  developed  that  correlates  interoperability 
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with  the  performance  criteria,  the  model  can  be  improved  through  multicriteria 
identification  to  a  level  of  adequacy  that  is  deemed  acceptable.  Statnikov  &  Matusov 
(2002)  define  the  experimental  value  of  the  vth  criterion  measured  from  the  prototype  as 
Ovexp  and  the  calculated  value  from  the  mathematical  model  as  0,c  to  derive  an 
expression  for  the  adequacy  criterion  as  follows: 

|  Ovc-Ovexp  |.  (4.1) 

For  our  purpose,  the  adequacy  criteria  would  be  a  measure  of  how  well 
correlated  the  performance  criterion  determined  by  the  interoperability  model  is  with  the 
actual  measurement  from  the  prototype  model.  Through  considerable  experimentation 
with  the  prototype  and  an  understanding  of  the  nature  of  the  measurement  error 
associated  with  the  performance  criterion  of  the  prototype,  a  maximum  value  of  the 
adequacy  criteria  can  be  determined  through  expert  analysis  and  in  notation  presented  by 
Statnikov  and  Mutusov  (2002),  written  as  an  inequality: 

|  d»vc_ovexp  |  <sv.  (4.2) 

The  value  of  sv  reflects  the  maximum  value  of  the  adequacy  criteria 
(desired  accuracy  of  correlation  between  physical  and  mathematical  models).  Examining 
the  variables  associated  with  the  models  that  still  satisfy  the  inequality  will  ensure  that 
alternative  representations  of  the  models  are  identified  that  may  prove  useful  specifically 
during  efforts  to  optimize  the  performance  criteria.  With  a  mature  mathematical  model 
and  high  degree  of  confidence  in  the  correlation  between  the  performance  criteria 
measure  predicted  by  the  interoperability  model  and  the  performance  measure  observed 
from  the  prototype,  the  mathematical  model  may  provide  a  cost  effective  and  rapid  means 
to  examine  the  SoS  for  potential  performance  improvement. 

b.  Use  of  Adequacy  Criteria  as  Baseline  Measure  for  C4  SoS 
Performance  Assessment 

The  process  required  to  develop  and  mature  a  mathematical  model  of  the 
SoS  also  provides  greater  insight  into  the  associated  performance  metrics.  With  an 
understanding  of  the  nature  of  the  measurement  error  associated  with  the  prototype, 
expected  values  for  the  performance  metrics  could  be  determined  and  used  as  a  basis  for 
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assessment.  The  inequality  (4.2)  could  serve  as  a  success  criteria  that  must  be  satisfied 
during  a  SoS  assessment.  If  an  assessment  results  in  not  meeting  the  criteria  (|  ®vc  - 
®vexp  |  >  sv),  further  evaluation  of  the  SoS  would  be  required  to  determine  if  a  change  to 
the  SoS  invalidated  the  mathematical  model  or  if  the  SoS  just  failed  to  perform  as 
expected. 

D.  SUMMARY 

Modeling  the  MAGTF  C4  SoS  is  an  important  facet  of  the  assessment  effort.  A 
representative  MAGTF  C4  SoS  that  serves  as  the  assessment  environment  is  an 
abstraction  of  the  actual  operational  SoS  environment  and  from  that  perspective  can  be 
considered  a  prototype.  This  prototype  can  serve  as  a  means  for  assessing  the  overall  C2 
effectiveness  of  the  SoS  if  appropriate  simulation,  stimulation  and  data  collection 
activities  are  employed  to  accommodate  both  mission  thread  execution  and  critical  C2 
background  activities.  A  complimentary  mathematical  model  would  also  add  value  by 
providing  a  capability  to  independently  predict  SoS  performance  behavior  and  gain  a 
better  understanding  of  where  and  how  to  best  optimize  the  prototype  environment  in 
pursuit  of  developing  a  more  effective  SoS  in  the  operational  environment.  The  i-Score 
methodology  provides  a  mathematical  model  that  relates  an  interoperability  score  to  the 
physical  attributes  of  the  SoS,  however  the  interoperability  score  would  be  significantly 
more  useful  if  it  can  be  correlated  with  performance  criteria  of  the  SoS.  The  use  of 
multicriteria  identification  may  provide  a  way  of  modifying  the  i-Score  model  (or  other 
mathematical  interoperability  model)  to  establish  a  correlation  between  the 
interoperability  score  and  relevant  performance  criteria.  If  developed,  the  mathematical 
model  could  provide  an  efficient  means  to  examine  modifications  to  the  SoS  in  pursuit  of 
performance  improvement.  Additionally,  the  use  of  adequacy  criteria  determined 
through  the  multicriteria  identification  process  may  serve  as  a  more  reasonable 
benchmark  for  evaluating  SoS  performance  during  an  assessment  event  by  providing  an 
expected  value  for  a  performance  criteria  that  accommodates  both  statistical  variance  in 
SoS  performance  behavior  and  aspects  of  known  measurement  error. 
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V.  CONCLUSIONS 


A.  KEY  POINTS  AND  RECOMMENDATIONS 

Over  a  number  of  years,  the  Marine  Corps  Tactical  Systems  Support  Activity  has 
aggressively  pursued  new  and  innovative  ways  to  improve  efforts  to  assess  the 
performance  of  tactical  C4  Systems  of  Systems  prior  to  deployment  in  an  operational 
environment.  In  that  pursuit,  determining  how  best  to  approach  performance  assessments 
of  large-scale,  complex  SoS  has  proven  a  challenge  that  extends  beyond  the  Marine 
Corps  to  a  fundamental  system  of  systems  engineering  challenges  shared  by  acquisition 
and  test  organizations  and  activities  throughout  the  DoD.  Selecting  key  performance 
criteria,  developing  methodologies  for  conducting  large-scale  SoS  assessments  and 
defining  more  quantitative  means  for  measuring  and  assessing  SoS  performance  and 
behavior  all  serve  as  a  central  challenge  and  the  genesis  of  the  initial  questions  that 
guided  this  research: 

1 .  How  can  large  scale  C4  System  of  Systems  be  tested  and  evaluated  in  a 
manner  that  reflects  performance  attributes  associated  with  the  System  of 
Systems  as  a  whole? 

2.  What  are  the  key  attributes  of  a  C4  SoS  that  may  serve  as  the  basis  for  SoS 
level  performance  criteria? 

3.  What  are  some  existing  assessment  tools  and  how  may  they  be  extended  to 
aid  in  C4  SoS  assessment  process? 

4.  How  might  multicriteria  identification  improve  or  contribute  to  the  Marine 
Corps’  C4  SoS  assessment  methodology? 

For  Joint  SoS  performance  assessments,  the  use  of  mission-based  test  threads 

serve  as  both  a  means  of  stimulating  the  SoS  and  a  means  for  assessing  performance  at 

the  SoS  level  (capability  of  the  SoS  to  execute  the  mission  thread  from  a  timeliness  and 

completion  percentage  rate).  The  use  of  a  mission-based  test  thread  is  also  extended  to 

MAGTF  C4  SoS  assessment  efforts,  but  with  a  specific  C4  focus,  other  attributes 

associated  with  providing  a  C2  capability  are  also  of  interest.  Incorporating  C2 
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background  activities  (situational  awareness,  position  reporting,  collaboration,  and  other 
DISN  services)  within  the  modeled  C4  architecture  provides  an  ability  to  assess  aspects 
of  the  SoS  with  respect  to  providing  a  C2  capability  (capability  of  the  SoS  to  provide  C2 
services  with  regard  to  bandwidth  efficiency  and  network  reliability). 

The  use  of  network  management  and  optimization  tools  that  are  typically  key 
components  of  operational  C4  SoS  have  not  typically  been  included  in  MAGTF  C4  SoS 
assessment  events  since  they  do  not  directly  contribute  to  execution  of  a  mission  thread. 
Adding  these  components  to  the  assessment  environment  provides  a  means  to  better 
model  QoS  and  bandwidth  availability  attributes  associated  with  the  SoS  and  may 
provide  additional  insight  into  and  measurement  of  performance  efficiency  from  a  C2 
background  activity  and  resource  utilization  perspective. 

In  reference  to  the  first  two  questions  that  guided  this  research  then,  it  is 
recommended  that  MAGTF  C4  SoS  assessment  efforts  include  aspects  of  providing  a  C2 
capability  in  addition  to  the  ability  of  the  SoS  to  execute  a  mission-based  test  thread  as 
the  basis  for  selection  of  SoS  level  performance  criteria.  This,  in  concert  with  addition  of 
network  management  and  optimization  tools,  will  provide  the  foundation  for  an 
assessment  with  focus  on  the  key  attributes  of  a  C4  SoS  from  a  more  holistic,  SoS 
performance  perspective. 

A  measure  of  SoS  interoperability  also  provides  a  useful  means  of  characterizing 

a  C4  SoS  in  terms  of  information  exchange  attributes  that  influence  both  the  efficiency 

and  reliability  of  a  SoS.  Guided  by  the  third  question  of  this  study,  three  assessment 

tools  were  reviewed  with  each  providing  a  unique  approach  for  evaluating  and  measuring 

SoS  interoperability.  The  most  promising  tools  for  continued  study  are  the  i-Score 

methodology  with  an  approach  that  provides  a  quantitative  measure  of  SoS 

interoperability,  and  the  DynSAVE  software  tool  that  provides  a  means  of  quantifying 

the  anomalous  behavior  and  relative  efficiency  of  the  SoS.  Based  on  this,  demonstrating 

and  further  evaluating  the  improved  i-Score  methodology  is  recommended  to  determine 

if  the  higher  level  of  fidelity  offered  by  the  improvements  add  to  or  enhance  the 

applicability  to  a  C4  SoS  assessment  event.  Demonstrating  and  evaluating  DynSAVE  is 

also  recommended  to  determine  whether  the  structural  and  behavioral  architecture 
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analysis  provided  by  DynSAVE  can  be  extended  to  provide  a  measure  of  efficiency  and 
reliability  during  a  MAGTF  C4  SoS  assessment.  DynSAVE  may  also  illustrate  specific 
data  exchange  attributes  between  SoS  components  that  may  help  determine  or  improve 
the  resemblance  coefficients  necessary  for  the  improved  i-Score  model. 

A  mathematical  model  that  determines  a  quantitative  value  for  SoS 
interoperability  like  that  offered  by  i-Score  may  certainly  complement  a  physical  model 
of  the  SoS  like  that  used  for  MAGTF  C4  SoS  assessments.  However,  the  interoperability 
measure  would  be  considerably  more  useful  if  it  can  be  correlated  with  SoS  performance 
criteria.  Guided  by  the  fourth  question  of  this  research,  multicriteria  identification  can 
contribute  to  the  Marine  Corps’  C4  SoS  assessment  methodology  by  providing  a  means 
to  modify  the  i-Score  model  (or  other  mathematical  interoperability  model)  to  establish 
that  correlation.  With  a  mature  mathematical  model  and  high  degree  of  confidence  in  the 
correlation  between  the  performance  criteria  measure  predicted  by  the  interoperability 
model  and  the  performance  measure  observed  from  the  prototype,  the  mathematical 
model  may  provide  a  cost  effective  and  rapid  means  to  examine  the  SoS  for  potential 
performance  improvement.  Additionally,  the  use  of  adequacy  criteria  determined 
through  the  multicriteria  identification  process  may  serve  as  a  more  reasonable 
benchmark  for  evaluating  SoS  performance  during  an  assessment  event.  For  those 
reasons,  pursuing  development  of  a  C4  SoS  mathematical  model  is  highly  recommended. 

Finally,  while  SoS  assessment  is  a  significant  challenge,  it  is  a  challenge  that  is 
shared  across  the  DoD  and  throughout  the  systems  engineering  community.  Continued 
involvement  in  Joint  DoD  efforts  such  as  JMETC  provide  the  Marine  Corps  with  a 
unique  opportunity  to  refine  assessment  methodologies,  examine  new  tools  and 
techniques  and  leverage  expertise  and  lessons  learned  through  continued  collaboration 
within  this  growing  community  of  interest.  Engagement  is  not  just  a  good  idea,  but  is 
essential  to  improve  and  mature  the  Marine  Corps’  engineering  and  test  effort  as  applied 
to  current  and  future  generations  of  the  MAGTF  C4  SoS. 
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B.  AREAS  TO  CONDUCT  FURTHER  RESEARCH 


A  number  of  areas  discussed  in  this  study  warrant  further  examination.  First, 
demonstration  and  evaluation  of  the  improved  i-Score  methodology  may  be  beneficial  to 
determine  if  the  improved  methodology  shows  potential  for  greater  applicability  to  C4 
SoS  assessments.  Second,  demonstration  of  the  DynSAVE  assessment  tool  during  a  C4 
SoS  assessment  would  provide  an  opportunity  to  examine  capabilities  and  limitations  of 
this  tool  in  a  controlled  environment.  Third,  further  study  towards  development  and  use 
of  a  mathematical  model  for  C4  SoS  assessment  and  optimization  may  provide 
considerable  benefit.  While  extensive  experimentation  through  a  physical  model  of  the 
C4  SoS  to  develop  and  improve  the  mathematical  model  is  necessary,  application  of 
simulation  and  stimulation  tools  and  virtualization  technology  may  help  expedite  this 
effort.  Finally,  virtualization  technology  may  serve  as  a  fruitful  area  of  study  from  two 
perspectives:  using  a  virtual  model  of  the  SoS  as  the  basis  for  an  assessment,  and 
determining  appropriate  tools  and  analysis  methodologies  for  assessing  C4  SoS  that 
employ  virtualization  technology  as  part  of  their  inherent  architecture. 

1.  C4  SoS  Assessment  Using  a  Virtual  Model 

The  use  of  virtualization  technology  to  model  the  C4  SoS  offers  considerable 
advantage  by  reducing  the  physical  resources  needed  to  create  the  C4  SoS  assessment 
environment.  The  benefits  of  using  a  virtual  model  to  validate  assessment  procedures  has 
been  demonstrated,  but  even  greater  value  would  be  derived  if  a  virtual  model  of  the  C4 
SoS  could  be  extended  to  serve  as  the  actual  assessment  environment.  Because  the 
virtual  model  presents  another  degree  of  abstraction  from  the  physical  model, 
experimentation  and  analysis  is  necessary  to  determine  where  and  how  variances  between 
the  virtual  and  physical  models  manifest  in  terms  of  SoS  performance  measures. 

2.  Assessment  Tools  and  Analysis  Concepts  for  C4  SoS  Analysis  in 
Virtual  SoS  Analysis  Environment 

A  number  of  Marine  Corps’  Systems  of  Systems,  including  the  COC,  intend  to 
employ  virtualization  technology  within  their  architectures.  Virtualization  of  network 
components  (servers  and  switches),  as  well  as  various  C2  client  components,  will  change 
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the  performance  and  behavior  attributes  of  the  SoS.  Additional  research  is  needed  to 
determine  whether  current  assessment  tools,  techniques  and  performance  criteria  are 
sufficient  to  conduct  an  assessment  of  a  C4  SoS  that  employs  virtualization  technology  or 
if  additional  or  new  tools  are  necessary. 
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